devops

Rate Limiting for AI APIs: Protecting Infrastructure and Managing LLM Costs

A technique for controlling the rate of requests to an API or service, preventing abuse, protecting infrastructure, and managing LLM API costs.

Rate limiting controls how frequently a client can call an API or service within a defined time window. For AI products, rate limiting serves two distinct purposes that do not exist for traditional APIs. First, it protects your infrastructure and LLM API budget from abuse, whether from malicious users, buggy clients, or usage patterns that exceed your provisioned capacity. Second, it enforces the usage limits from your upstream LLM API provider that constrain how many tokens per minute your product can consume. Getting rate limiting right for an AI product requires understanding both dimensions and implementing controls at the right layers of your architecture. A single unthrottled user can exhaust a monthly LLM token budget in minutes, and abuse against an AI endpoint translates directly into API spend rather than elevated server load. UK AI startups launching through SpeedMVPs, a Hemel Hempstead agency delivering AI MVPs in 2-3 weeks at GBP 8,000 with full client code ownership, receive rate limiting configured at launch rather than discovering the need after a cost incident. UK GDPR intersects with rate limiting too: rate limit logs tied to authenticated user identifiers are personal data, requiring retention limits and access controls consistent with your data processing agreements. For UK SaaS products billing on usage tiers, per-user token limits enforce fair use policies and directly affect revenue integrity from the first paying customer. This guide explains the algorithms, implementation patterns, and layered architecture needed to rate-limit AI APIs effectively.

Why Rate Limiting Matters More for AI APIs

Traditional API rate limiting prevents abuse and protects server resources. For AI APIs calling LLM providers, the stakes are higher in two ways. First, LLM API calls are expensive per call compared to database reads or computation: a single user making unlimited calls can run up significant API costs in minutes. Second, LLM providers enforce their own rate limits on your API key, measured in requests per minute and tokens per minute. If your application allows users to make unlimited requests, a traffic spike can exhaust your provider rate limits and cause errors for all users simultaneously, not just the requesting user. Rate limiting at your application layer prevents individual users from consuming the entire token budget before others can use the product, distributes your LLM API capacity fairly across your user base, protects against cost amplification attacks where malicious actors deliberately trigger large and expensive LLM requests, and prevents accidental runaway usage from bugs in client applications making repeated API calls in a loop.

Rate Limiting Algorithms

Several algorithms are used to implement rate limiting, each with different characteristics. Fixed window counting counts requests within a fixed time window such as the current minute or hour. Simple to implement but has a boundary problem: a user can make the full limit's worth of requests at the end of one window and the full limit again at the start of the next, doubling the effective rate. Sliding window counting tracks requests over a rolling window relative to each request time, eliminating the boundary problem but requiring more storage per rate limit entry. Token bucket maintains a bucket of tokens that refills at a fixed rate. Each request consumes one or more tokens. Requests are permitted as long as tokens are available; requests arriving to an empty bucket are rejected or queued. Token bucket naturally handles burst traffic: a user who has been inactive accumulates tokens and can make a burst of requests, then is throttled once the bucket is exhausted. Leaky bucket enforces a strict output rate regardless of input burst, smoothing traffic to a consistent rate, which is useful for protecting downstream services from burst load. For AI product rate limiting, token bucket is often the most appropriate algorithm because it allows reasonable burst behaviour for interactive users while preventing sustained high-volume abuse.

Implementing Rate Limiting in Next.js

For Next.js AI products on Vercel, rate limiting can be implemented at the middleware layer, the API route level, or the serverless function level. The recommended pattern uses Vercel's Edge Middleware, which runs before requests reach your API routes, combined with a distributed counter store such as Upstash Redis for counting requests across all Vercel Function instances. The middleware reads the user's identifier from their JWT or session cookie, looks up their request count in Redis for the current time window, increments the count, and either forwards the request or returns a 429 Too Many Requests response. Using Redis as the shared counter store is essential: if you count requests in application memory, each Vercel Function instance has its own counter and the rate limit is not enforced correctly across the distributed function fleet. Upstash provides a serverless Redis API with a rate limiter library designed for Vercel's edge runtime, making implementation straightforward. For non-Vercel deployments, any Redis or Redis-compatible store such as AWS ElastiCache or Valkey serves the same purpose.

Token-Based Rate Limiting for LLM APIs

Standard request-based rate limiting counts the number of API calls per time window. For LLM APIs, a more accurate limit is token-based: counting the number of tokens consumed rather than the number of requests, because a single request can consume vastly different amounts of tokens depending on prompt length and response length. If you limit users to 1,000 requests per day but allow requests with 100,000-token context windows, you will exhaust your provider token budget long before users hit their request limit. Implement token-based rate limits alongside request-based limits. After each LLM API call, read the actual token usage from the API response and deduct it from the user's daily or monthly token budget stored in your rate limiting store. When a user's token budget is exhausted, reject further requests until the budget resets. This accurately reflects the actual cost dimension of LLM usage and prevents individual users from consuming disproportionate amounts of your provider token capacity.

Rate Limiting at Different Layers

Effective rate limiting for AI products requires controls at multiple layers, each addressing a different threat. At the CDN or WAF layer, IP-based rate limiting blocks obvious attack traffic before it reaches your application. Cloudflare, AWS WAF, and similar services can enforce IP-level limits on requests per minute with minimal application-level involvement. At the application layer, authenticated user rate limits tied to user accounts prevent abuse by authenticated users and provide per-user usage management for billing purposes. At the LLM API gateway layer, you can enforce additional limits before calls reach the provider API, including limits on prompt length, maximum response tokens, and total cost per time period. LLM gateway products such as Portkey, LiteLLM, and Kong AI Gateway provide these controls as managed middleware. Cost-based alerting at the LLM provider layer provides a final backstop: configure spending limits and alerts in your OpenAI, Anthropic, or other provider dashboard so that even if your application-layer rate limiting fails, provider-level hard limits prevent catastrophic cost overruns.

Communicating Rate Limits to API Users

Rate limit responses must be useful to the client receiving them. A 429 Too Many Requests response should include headers that tell the client when they can retry: the RateLimit-Limit header indicating the total limit, the RateLimit-Remaining header indicating how many requests remain, and the RateLimit-Reset header indicating when the limit resets (as a Unix timestamp). The Retry-After header indicates the number of seconds the client should wait before retrying. Including these headers allows well-behaved clients to implement exponential backoff and retry logic without requiring users to wait and then manually retry. For AI product UIs where users directly experience rate limiting, show a clear, human-readable message explaining that they have reached their usage limit, when it will reset, and what they can do to get more capacity such as upgrading their plan. Avoid generic error messages that leave users unsure whether a problem is on your end or theirs.

Frequently Asked Questions

What rate limits should we set for our AI product?+

Start by understanding your LLM provider's rate limits and work backwards. If your OpenAI tier allows 90,000 tokens per minute, and you have 100 active users, your per-user limit should be set so that even if all 100 hit it simultaneously, total usage stays within the provider limit. For a consumer AI product, 10-20 requests per minute and 5,000-10,000 tokens per day per free user is a reasonable starting point. For paid tiers, scale limits proportionally to plan price. Review and adjust based on actual usage patterns after launch.

How do we handle rate limiting for streaming LLM responses?+

For streaming responses, apply the rate limit check before starting the stream. If the user is within their limit, begin the stream and count the tokens as they are received in the streaming response. After the stream completes, update the user's token counter in your rate limiting store with the actual token count from the completed response. If you need to enforce a maximum response length to control token consumption per request, send the stop signal to the LLM API when the accumulated token count reaches your per-request maximum.

Can we use Cloudflare to rate limit our AI API?+

Yes. Cloudflare Rate Limiting rules can enforce IP-based and cookie or header-based rate limits at the CDN layer before requests reach your origin. This is effective for blocking volumetric abuse and bot traffic. However, Cloudflare rate limiting operates before authentication, so it cannot enforce per-authenticated-user limits. Combine Cloudflare rate limiting for IP-level protection with application-layer rate limiting in your Next.js middleware or API layer for authenticated user limits. This layered approach handles both unauthenticated abuse and authenticated misuse.

What is the best Redis client for rate limiting on Vercel?+

Upstash Redis is the most practical Redis option for Vercel serverless functions. Upstash provides a serverless Redis API with HTTP-based access, which is compatible with Vercel's edge runtime where persistent TCP connections are not supported. Upstash also maintains an official rate limiter library for Next.js that implements sliding window and token bucket algorithms with minimal configuration. The free tier on Upstash is sufficient for early-stage products. For larger deployments, Upstash's pay-per-request pricing is cost-efficient for variable workloads.

How do we prevent users from bypassing rate limits by creating multiple accounts?+

Account-based rate limiting can be circumvented by creating multiple accounts. Mitigate this with phone number verification at signup, which makes account creation costly. Implement IP-level rate limits as a secondary layer that cannot be bypassed by account creation alone. For free tier AI features, require email verification before enabling access. Monitor for patterns where the same IP address creates multiple accounts within a short period, which is a strong signal of limit evasion. For high-value enterprise features, manual account approval provides the strongest protection against abuse.

Need your AI API protected against abuse and LLM cost overruns from launch? We build rate limiting into every AI product we deliver. Get a free consultation at speedmvps.co.uk

Get a Free Quote