What's Happening?
Meta has introduced the next generation of its Andromeda system, a personalized ads retrieval engine, which now processes over one billion ads per second. This significant throughput increase, a 100x lift compared to the 2024 Meta Andromeda system, is attributed
to a co-design approach integrating model, system, and GPU hardware. The updated Andromeda system incorporates three architectural innovations, including Large Language Model (LLM)-inspired Mixture-of-Experts (MoE) into the retrieval process. This new architecture allows for near-ranking-level performance at approximately 300x fewer inference-time FLOPs than its predecessor. The system also features a single GPU kernel that fuses both model and system workloads per user-ad interaction, and a co-location strategy that places index matching, scoring, and top-K selection on the same GPU. This eliminates network and memory bottlenecks, which had become critical at the multi-billion ads per second scale. The advancements are a result of a joint collaboration with NVIDIA, optimizing GPU kernels and the full retrieval flow.
Why It's Important?
This technological leap by Meta has substantial implications for the digital advertising industry and the broader economy. For advertisers, it means more relevant ads delivered to users, leading to better customer matches and increased conversions. Meta reports a 2.4% increase in conversions on Facebook since 2025 due to Andromeda. For Meta, this translates to increased business value from advertisers and reduced operational costs, as each GPU can now evaluate more ads per second. The ability to scale ads retrieval without proportional infrastructure growth is crucial as the volume of eligible ads continues to expand, driven by generative AI creative tools. This innovation addresses the challenge of processing an unprecedented scale of ad candidates efficiently, ensuring that small improvements in retrieval model performance can drive significant gains in advertiser value and Meta's revenue. The co-design principle, integrating hardware and software, sets a new benchmark for efficiency in large-scale machine learning systems.
What's Next?
Meta plans to continue its optimization and co-design efforts with NVIDIA, extending this work to next-generation NVIDIA platforms like Grace Blackwell. The focus will be on platform design optimizations that allow for different CPU-to-GPU compute ratios, which are considered important for Andromeda-like workloads. Further evaluations will assess the sensitivity of end-to-end application performance to these ratios, utilizing the performance capabilities of Grace-class CPUs and Blackwell-class GPUs. The ongoing work aims to explore how these findings can be applied to future platform generations, potentially bringing more architectural improvements to Meta's ad retrieval systems. The current architecture, with its significant headroom, is positioned to absorb greater model complexity, including deeper interaction networks, richer feature spaces, and additional retrieval pathways, without requiring proportional increases in infrastructure. This suggests a continuous evolution towards more sophisticated and efficient ad delivery mechanisms.
Beyond the Headlines
The advancements in Meta's Andromeda system highlight a broader trend in the technology industry: the increasing importance of model-system co-design for achieving breakthroughs in performance and efficiency. This approach, where hardware and software are developed in tandem, is becoming critical for managing the immense computational demands of modern AI and machine learning applications, particularly in areas like personalized recommendations and advertising. The ethical implications of highly efficient and personalized ad retrieval systems are also significant. While more relevant ads can improve user experience, they also raise concerns about data privacy, algorithmic bias, and the potential for sophisticated targeting to influence user behavior in subtle ways. The ability to process billions of ads per second with near-ranking-level accuracy means that the system can exert a powerful influence on what information and products users encounter, necessitating ongoing scrutiny of its societal impact and the responsible development of such powerful AI technologies.
