More Than Just Requests Per Minute
When most developers think about API limits, they think of one number: requests per minute (RPM). It’s a simple concept, like a turnstile that only lets a certain number of people through every 60 seconds. If your limit is 60 RPM, you can make one call to the API every second. For small projects and testing, this is usually the only metric that matters. You write your code, stay under the limit, and everything works. But OpenAI, like other advanced AI providers, operates on a two-part system. The second, and far more critical, constraint is Tokens Per Minute (TPM). A token is a piece of a word, roughly equivalent to four characters of text. Every piece of information you send to the model (the prompt) and receive back (the completion) is measured
in tokens. This means it’s not just about how many times you call the API, but how much “work” you’re asking it to do in each call.
The Two-Constraint Problem
Here is the detail that trips up entire engineering teams: you must stay under *both* your RPM and TPM limits. Whichever one you hit first slams the brakes on your application. This is the crucial nuance. You might have a generous RPM limit of 3,500, but a much tighter TPM limit of, say, 300,000 on a new, powerful model. A developer might look at their RPM and think, “Great, I have plenty of room.” They are wrong. With the latest, most capable models like GPT-4o, both prompts and responses can be extremely long and complex. A single request to summarize a large document, analyze a complex dataset, or generate a detailed report could easily consume 10,000, 20,000, or even more tokens. In this scenario, just 15 of these “heavy” requests in a minute would exhaust your 300,000 TPM limit, even though you’ve only used a tiny fraction of your 3,500 RPM allowance. The API will start returning error messages, not because you’re making too many calls, but because your calls are too demanding.
Why This Cripples Apps at Scale
This TPM bottleneck is what makes an otherwise functional application “unusable at scale.” During development, with one or two users, you’ll almost never hit the token limit. The application feels fast and responsive. But the moment you launch to hundreds or thousands of users, the system collapses. It’s not a linear scaling problem; it’s a sudden, catastrophic failure. Imagine a customer service bot built on the latest model. Ten customers are using it simultaneously. One asks for a detailed summary of their entire year’s worth of support tickets. That single, legitimate request devours a massive chunk of the minute’s token budget. For the other nine customers, the bot suddenly stops working. They get error messages or endless loading spinners. From their perspective, the service is broken. The business just created a terrible user experience, not because of a bug in the code, but because of a fundamental misunderstanding of the platform’s architecture.
Navigating the Token Bottleneck
Experienced developers don’t fight the rate limits; they engineer around them. The solution isn’t to simply ask OpenAI for a higher limit, though that’s part of it. The real work involves building a more intelligent system. First is implementing a robust queuing system. If a request would exceed the TPM limit, it’s not sent. Instead, it’s placed in a queue and retried after a delay. This smooths out spikes in demand. Second is prompt optimization. Engineers work to get the same quality of response using fewer tokens, treating them like a precious resource. Third, for high-throughput systems, is load balancing across multiple API keys, each with its own rate limit. Finally, it involves building sophisticated monitoring to track both RPM and TPM usage in real-time, so the system can predict and adapt to bottlenecks before they occur. It transforms the problem from a simple API call into a complex resource management challenge.
