Choosing a Model & Provider

The plain-language idea

Picking a model for a feature isn't a single decision, it's a tradeoff across several axes at once:

• Cost per token — providers price models very differently; a more capable model can cost many times more per token than a smaller one.
• Latency — smaller models generally respond faster, which matters a lot for anything user-facing and synchronous (autocomplete, a chat reply someone is waiting on).
• Capability for the specific task — this is the one people most often get wrong. Classification, extraction, formatting, and short factual lookups are frequently well within reach of a small, cheap, fast model. Multi-step reasoning, ambiguous judgment calls, or long-context synthesis are where a stronger (and pricier, slower) model actually earns its cost.
• Context window size — how much input a model can accept in one call; a feature that needs to reason over a large document or long conversation history needs a model with room for it, regardless of how "smart" it is.
• Vendor lock-in risk — writing application code that calls one provider's SDK directly, with that provider's exact request/response shapes baked in everywhere, makes switching providers later expensive. This is part of why an abstraction layer (a shared interface your code calls, with the actual provider swapped in behind it) matters, beyond just being "clean architecture."

The framing worth internalizing: match the model to the task's actual difficulty, not "always use the most capable model available." Running every request, including trivial ones, through an expensive frontier model is a real and common source of waste — it's slower and costlier for no quality benefit on tasks a cheaper model already handles correctly.

See it in a real thing: one feature, two very different tasks

Task A: classify a support ticket into one of 6 categories
  -> short input, short output, well-defined labels
  -> a small, fast, cheap model gets this right essentially every time

Task B: read a 40-page contract and flag ambiguous liability clauses
  -> long input, requires genuine reasoning and judgment
  -> worth paying for a stronger model here

Both tasks might live in the same product, even the same codebase, but routing both through the same "best available" model means overpaying (and adding latency) on Task A for no quality gain, while correctly needing that spend on Task B. A codebase built with a provider-agnostic entry point — this app's own lib/ai/chat/handler.ts is exactly that shape, a single function every AI call site goes through rather than each call site talking to a provider's SDK directly — makes it straightforward to route different tasks to different models, and to swap providers later, without rewriting every call site.

What people usually miss

Watch out

• A bigger context window doesn't imply a smarter model, or vice versa. These are separate axes — a model can have a large context window and modest reasoning ability, or the reverse. Pick based on which axis your task actually stresses.
• Latency compounds in multi-step or agentic flows. A single extra second per call feels trivial once; across a chain of several sequential model calls it becomes a real, user-visible delay — task-appropriate model choice matters more, not less, as call chains grow.
• "Good enough" is a legitimate, deliberate target, not settling. For a well-scoped task like classification or structured extraction, a smaller model matching or nearly matching a larger model's accuracy at a fraction of the cost is the correct outcome to design for, not a compromise.
• Vendor lock-in is a real switching cost, not a hypothetical one. Providers change pricing, deprecate models, and have outages — code that only knows how to talk to one provider's exact API shape makes reacting to any of that far more expensive than code built behind a shared interface.

Why this matters in interviews

"How would you decide which model to use for a new AI feature?" is a real question, and naming cost, latency, task-appropriate capability, context window, and lock-in — rather than reflexively answering "the most powerful one available" — is what separates someone who's actually shipped AI features from someone repeating marketing copy.