AI Feature Rollout & Fallbacks

The plain-language idea

An AI call is a network request to a system you don't control, that can be slow, wrong, rate-limited, or briefly down, on a schedule you have no visibility into. Shipping an AI feature to production means designing for that reality up front, not discovering it the first time a provider has an outage. Three patterns do the real work:

• Graceful degradation. When an AI call fails or times out, the UI shouldn't just break. A cached or canned fallback response, a clear "try again" state, or falling back to a non-AI code path entirely (a plain search instead of an AI answer, a static template instead of a generated one) all beat a broken screen or an infinite spinner.
• Feature-flagging the AI feature itself. Wrapping the feature behind a flag (the generic mechanism from the feature-flags lesson) means it can be killed instantly, with no deploy, if a model starts behaving badly, a provider has an incident, or costs spike unexpectedly. This is specifically valuable for AI features because the failure modes (a model quietly getting worse, a cost spike from unexpected usage patterns) are less predictable than a typical code bug, so having an instant kill switch matters more here than for most features.
• Provider fallback chains. If the primary provider errors out or is rate-limited, retry the same request against a secondary provider instead of failing the whole request. This is the same shape as the BYO-key fallback ladder from the earlier lesson, just one layer up: instead of falling back from a platform key to a user key on the same provider, you fall back from one provider to another entirely.

None of these are exotic. They're the same defensive-engineering instincts used everywhere else in a production system — timeouts, retries, circuit breakers, kill switches — applied specifically to the AI call site, because that call site fails in its own particular ways (rate limits, quota exhaustion, occasional bad output) that a generic API client doesn't usually have to think about.

See it in a real thing: this app's own defensive chain

// lib/ai/chat/handler.ts
export async function handleChat(messages, deps) {
  const primary = resolveProviderConfig(deps); // platform key -> user key -> NeedsUserKeyError
  try {
    return await make(primary).chat({ messages });
  } catch (err) {
    if (err instanceof QuotaExhaustedError) {
      const user = deps.userConfig();
      if (user && user.apiKey !== primary.apiKey) {
        return await make(user).chat({ messages }); // fall back and retry
      }
    }
    throw err; // caller surfaces a real, actionable error state
  }
}

This one function is the whole track in miniature: resolveProviderConfig (the BYO-key lesson) picks a config; the actual call goes through a single shared entry point rather than each call site touching a provider directly (the model-choice lesson's abstraction-layer point); a live quota failure triggers a real fallback retry instead of surfacing a broken UI (this lesson's degradation pattern); and every call site that wants observability (the observability lesson) has exactly one place to add it. None of these pieces is sufficient alone — the fallback is only useful because there's a real config to fall back to, and the whole thing is only debuggable in production because it's logged.

What people usually miss

Watch out

• A provider fallback chain and a BYO-key fallback are the same underlying shape at different layers. One falls back from a platform key to a user's own key on the same provider; the other falls back from one provider to a different one entirely — recognizing they're the same pattern applied twice makes both easier to reason about and to test.
• Graceful degradation needs a real fallback experience, not just a caught error. Catching the exception so the app doesn't crash is necessary but not sufficient — the user still needs something coherent to see (a canned response, a retry affordance, a non-AI path), not a swallowed error and a blank space.
• A kill switch only helps if someone's actually watching for the signal to flip it. A feature flag without monitoring/alerting on cost or error-rate spikes is a switch nobody knows to flip in time — the flag and the observability from the earlier lesson are meant to work together, not substitute for each other.
• Fallback logic needs to be tested, including the failure path, not just the happy path. A fallback chain that's never actually exercised (because the primary provider has never failed in dev or staging) is unverified code exactly where correctness matters most — under real failure conditions.

Why this matters in interviews

"Walk me through how you'd make an AI feature production-ready, not just a working demo" is a real, common closing question for AI-feature work, and a strong answer ties together cost control (caching, model choice), debuggability (observability), and resilience (degradation, flags, provider fallback) as one coherent system — not a list of unrelated nice-to-haves.