Rate Limiting & Cost Control for AI Features

The plain-language idea

This track's Frontend Practices lesson on rate limiting already covers the generic mechanism: capping how many requests a client can make in a window. That mechanism still applies to AI features, but it's not the whole story — AI calls have a cost dimension a typical API call doesn't. Most APIs cost roughly the same per request regardless of payload size. An AI API bills per token, counting both the input you send (system prompt, conversation history, retrieved context) and the output the model generates — so cost scales with two things a request-count limit alone doesn't control: how much context goes in, and how long the response is allowed to run.

That distinction matters because a single feature can blow up cost without ever exceeding a request-count rate limit. The most common way this happens: an unbounded conversation, where every new turn re-sends the entire prior history as context. Turn one sends a small prompt; turn fifty sends fifty turns' worth of accumulated history, every single time, at every single turn. The request count didn't change; the cost per request quietly grew the whole time.

Three real controls address this directly:

• Per-user/per-session quotas — specifically to prevent abuse, capping how many expensive completions any one user can generate in a period, independent of generic request-rate limiting.
• Max output tokens per request — a hard ceiling on how long any single response is allowed to run, so one prompt can't produce an unexpectedly long (and expensive) generation.
• Truncating or summarizing conversation history — instead of sending the full unbounded transcript every turn, trim to the most recent N turns or periodically summarize older turns into a shorter form, keeping the input side of the cost bounded as a conversation grows.

See it in a real thing: an unbounded chat history

Turn 1:  send [system, msg1]                          -> ~200 input tokens
Turn 10: send [system, msg1..msg10]                    -> ~2,000 input tokens
Turn 50: send [system, msg1..msg50]                    -> ~10,000 input tokens

Nothing here looks like abuse. It's one engaged user having one long, legitimate conversation — and the cost per turn grew roughly linearly the entire time, purely because every turn resent everything before it. A per-request rate limit wouldn't catch this at all, since the number of requests never changed; only fixing the shape of what each request sends (truncation, summarization, or a sliding window of recent turns) actually bounds it.

What people usually miss

Watch out

• Cost scales with output length too, not just input size. A max-output-tokens cap is a cost control, not just a UX nicety — an unbounded generation is an unbounded bill even if the input side is well-managed.
• A request-count rate limit doesn't bound cost by itself. Ten requests at 500 tokens each and ten requests at 50,000 tokens each are both "ten requests" — request-count limiting and token-cost control are two different, complementary safeguards, not one general rate limiter covering both.
• Truncating history is a real product tradeoff, not a free optimization. Dropping or summarizing older turns risks losing context the model actually needed — the right window size or summarization strategy has to balance cost against the conversation staying coherent.
• Per-user quotas exist specifically for abuse, not just fairness. Without one, a single user (accidentally, via a retry loop, or deliberately) can generate unlimited expensive completions against your account with no generic-rate-limit rule catching it, since abuse here is about accumulated cost, not request frequency.

Why this matters in interviews

"How would you stop an AI feature's costs from running away in production?" tests whether a candidate reaches specifically for token-aware controls (output caps, history truncation, per-user quotas) rather than just restating generic rate limiting, which alone doesn't bound per-token cost.