In the world of AI, it feels intuitive: more power equals more speed. So, if you want faster answers from a model like Claude, you just add more GPUs, right? Not so fast. The reality is far more complex, and often, the real bottleneck isn't raw compute.
The Core of the Misconception
Graphics Processing Units (GPUs) are the undisputed workhorses of the AI world. Their ability to perform thousands of parallel calculations at once makes them perfect for the massive matrix multiplications at the heart of large language models (LLMs) like those from Anthropic. The common assumption is that if one GPU is good, two must be twice as good for getting a single answer back faster. This line of thinking, however, confuses two very different performance metrics: throughput and latency.
Latency vs. Throughput: The Highway Analogy
Imagine a highway. Throughput is the number of cars that can pass a certain point per hour. Latency is the time it takes for one specific car to travel from its start to its destination. Adding more GPUs is like adding more lanes to the highway. It dramatically increases throughput, allowing you to handle many more simultaneous requests from different users. If you're a company serving thousands of users at once, this is fantastic. However, adding lanes doesn't necessarily raise the speed limit. The time it takes for a single car—or a single Claude response—to complete its journey from start to finish (its latency) might not change at all.
The Real Bottleneck: Memory Bandwidth
When an LLM like Claude generates a response token by token, the process is often 'memory-bound,' not 'compute-bound'. Each new token requires the model to read its massive set of parameters (the weights) from the GPU's high-bandwidth memory (HBM). For a large model, this can involve moving hundreds of gigabytes of data. The GPU's powerful processing cores can sit idle, waiting for this data to arrive. This is known as the 'memory wall'. No matter how much computational power you add, you can't generate the next token any faster than the memory system can deliver the necessary weights. It's like having the world's fastest chef who has to wait five minutes for every single ingredient to be delivered to their station.
It's a Whole-System Problem
Beyond memory bandwidth, other factors limit single-response speed. Generating a response involves two phases: 'prefill', where the initial prompt is processed, and 'decode', where the answer is generated token by token. Prefill is compute-intensive and benefits from more processing power, but the decode phase is often the slowest part and is bound by memory speed. Furthermore, the entire system—from how requests are scheduled and batched to the efficiency of the software managing the process—plays a huge role. Even the physical interconnects that link multiple GPUs can become a bottleneck if they aren't fast enough to handle the communication required for certain types of parallelism.
When More GPUs Do Reduce Latency
This isn't to say adding GPUs never helps with speed. For models that are too massive to fit into a single GPU's memory, splitting the model across multiple GPUs is a necessity. In this case, using a technique called 'tensor parallelism', where different GPUs work on separate chunks of the same calculation, can indeed reduce latency—provided they are connected by extremely fast links like NVLink. This is different from simply running two separate copies of the model. Companies like Anthropic are even designing their own custom chips to optimize this hardware-software dance and boost Claude's efficiency directly.
