Throughput: The GPU's Raw Horsepower
Think of a Graphics Processing Unit, or GPU, as a massive factory designed for one thing: parallel work. Unlike a CPU that handles a few complex tasks sequentially, a GPU has thousands of smaller cores that perform many calculations simultaneously. This
is its 'throughput'—the sheer volume of computational work it can get done at any given moment, often measured in FLOPS (floating-point operations per second). For AI, this is crucial. Training a large language model involves bombarding it with data and having it perform countless calculations in parallel to learn patterns. Higher throughput means more calculations at once, which can drastically shorten training times. This is the raw horsepower that companies like Microsoft are paying a premium for when they buy top-tier chips from Nvidia.
Memory Bandwidth: The Data Superhighway
Now, imagine that factory has thousands of workers (the cores) ready to go, but the conveyor belts supplying them with parts are slow. The factory's output will plummet. That conveyor belt speed is the GPU's 'memory bandwidth'. It measures how quickly the GPU can move data from its dedicated memory (VRAM) to its processing cores and back again. An AI model's parameters, or 'weights', are the data that needs to be constantly accessed. If bandwidth is low, the powerful processing cores sit idle, waiting for data. For generating AI responses—a process called inference—bandwidth is often the primary bottleneck. The model has to retrieve the right weights from memory to generate each new word or 'token'. So, while throughput is the potential to do work, bandwidth is the ability to feed that potential.
The Billion-Dollar Balancing Act
For AI workloads, throughput and bandwidth are in a delicate, expensive dance. You can't have one without the other. A GPU with massive throughput but insufficient bandwidth is like a Ferrari engine stuck in city traffic—all that power is wasted. Conversely, massive bandwidth feeding a weak processor is equally inefficient. The goal for AI companies is to keep their multi-thousand-dollar GPUs fully utilized, which means finding the perfect balance between the two for a given task. An imbalance means you are paying for idle silicon, a cardinal sin when a single high-end Nvidia H100 GPU can cost over $3 per hour to rent from a major cloud provider like Microsoft Azure.
Inside Microsoft's AI Cost Structure
This brings us to Microsoft. The company is planning to spend hundreds of billions on AI infrastructure through 2030, with capital expenditures for 2026 alone estimated at around $175 billion. A huge portion of this goes toward acquiring and operating massive fleets of GPUs for its Azure cloud service. Customer demand for AI services on Azure is reportedly exceeding Microsoft's available capacity. The company's challenge is to turn this colossal investment into profit. CEO Satya Nadella has emphasized focusing on the 'cost-to-outcome' frontier, meaning customers will measure the cost of using AI against the value it produces. To make services like Microsoft 365 Copilot commercially viable at scale, Microsoft must relentlessly optimize its use of GPU throughput and bandwidth to drive down the cost of generating each answer. Every fraction of a cent saved per query, multiplied by millions of users, translates into enormous sums, justifying the immense upfront hardware cost.
