AI Observability & Logging

The plain-language idea

The generic observability principle — logging discrete events, monitoring aggregate metrics, being able to answer a new question about system behavior after the fact without shipping a new debug build — applies to AI features too, but a normal API log doesn't capture what actually matters for an LLM call. Four things are specifically worth tracing:

• The actual prompt sent — not a summary of it, the real text (or messages array) that went to the model. This is what lets you reconstruct why a response went wrong later.
• Token counts — input and output tokens per call, which is what AI cost is actually billed on; without this, cost tracking and budgeting is guesswork.
• Latency, broken down by time-to-first-token vs. total completion time — these are meaningfully different for a streaming response. A slow time-to-first-token means the user stares at nothing before text starts appearing; a slow total time means the full response takes a while even once it starts streaming. Conflating them into one "response time" number hides which one is actually the problem.
• Which model/provider actually handled the request — essential once there's any fallback logic across providers or models, since "the AI feature is slow/wrong" is a very different bug depending on whether the primary or the fallback provider served that particular request.

The reason this matters specifically for AI calls, more than for a typical deterministic API: LLM output is non-deterministic. The same prompt sent twice can produce two different responses. That means "it happens sometimes" is not debuggable on its own — you need the actual historical prompt and the actual historical response for the specific bad output someone reported, not just a general sense that the feature is occasionally wrong. Without that concrete record, there's nothing to diagnose against.

See it in a real thing: tracing a single chat call

const start = performance.now();
const result = await handleChat(messages, deps);
const totalMs = performance.now() - start;

logger.info("ai_chat_call", {
  userId: user.id,
  provider: result.provider,       // which provider actually served this
  model: result.model,
  promptTokens: result.usage.promptTokens,
  completionTokens: result.usage.completionTokens,
  timeToFirstTokenMs: result.timing.ttft,
  totalMs,
  // the actual prompt/response pair — this is the part a generic API log skips
  prompt: messages,
  response: result.text,
});

If a user later reports "the tutor gave me a wrong explanation," a generic log ("chat request, 200 OK, 1.2s") gives you nothing to work with. This log gives you the exact prompt that produced that exact response — which is the only way to actually investigate a single bad output from a system that won't necessarily reproduce it if you just ask the same question again yourself.

What people usually miss

Watch out

• "It's non-deterministic" is not an excuse to skip logging — it's the reason logging matters more here. Because the same input can produce different outputs, you can't just re-run a prompt to reproduce a reported bad response; the historical record is often the only evidence you'll get.
• Time-to-first-token and total completion time are different metrics, not one metric measured two ways. A feature can have fast TTFT and slow total time (long response, user sees it start quickly) or the reverse (long delay before anything appears, then a fast stream) — they point to different problems and both need tracking separately for a streaming feature.
• Token counts are the actual cost signal, not request count. Two requests can cost wildly different amounts depending on prompt length and response length — logging request counts alone tells you volume, not spend.
• Model/provider attribution only matters once there's a fallback path. A single-provider feature doesn't need this field to debug quality issues, but the moment there's fallback logic (like this app's own provider fallback in lib/ai/chat/handler.ts), not knowing which provider actually served a given request makes a quality regression impossible to localize.

Why this matters in interviews

"A user says your AI feature gave a bad answer — how do you debug it?" is a real question, and the honest answer requires having actually logged the specific prompt/response pair, token usage, and latency breakdown ahead of time — not "we'd look into it," which isn't possible without that historical record already existing.