Model Selection

There is no single best model. There is a best model for each task at an acceptable cost and latency. An assistant usually uses three to five models: a strong one for answers, small fast ones for rewriting and classification, an embedding model, and a re-ranker.

Match models to tasksCopy Anchor Link

Model tiers move quickly. A small model from this year often beats a large model from two years ago. Re-run your evaluation set when new models appear rather than relying on benchmark leaderboards, which rarely match your tasks.

How to chooseCopy Anchor Link

1. Build the evaluation set first. 100 to 300 real questions with expected answers or grading criteria. See Datasets.
2. Run every candidate model on the same set with the same prompt, adapted only where a model requires it.
3. Score quality, latency, and cost together.

4. Pick the cheapest model that meets your quality bar. If Model B is within the margin of error of Model A at half the cost, choose B.
5. Check the failure cases, not just the average. A model with a slightly higher average score but more confident wrong answers is worse for an enterprise assistant.

Also check things benchmarks ignore: behavior in each of your users' languages, refusal rate on legitimate work questions, and how well it follows your citation format.

Non-functional requirementsCopy Anchor Link

Model gatewayCopy Anchor Link

All model calls go through one internal component, the model gateway. It can be a library inside your backend or a separate service. It exists so that feature code never depends on a specific provider.

ResponsibilitiesCopy Anchor Link

Logical model namesCopy Anchor Link

Feature code should never contain "gpt-x-2026-01" or similar. It asks for a role:

# model-routes.yaml
answer:
  primary: provider_a/large-2026-02
  fallback: [provider_b/large-latest, provider_a/medium-2026-02]
  max_output_tokens: 2000
fast:
  primary: provider_a/small-2026-02
  fallback: [provider_b/small-latest]
judge:
  primary: provider_b/large-latest
embed:
  primary: provider_a/embed-v3   # changing this requires re-indexing

Changing the model behind answer is then a config change, rolled out with a canary and evaluated, not a code change across many files.

Embeddings are the exception. You cannot fall back to a different embedding model at query time, because the query vector must come from the same model as the indexed vectors. Changing the embedding model means re-embedding the whole corpus. See RAG ingestion.

Routing strategiesCopy Anchor Link

Start with static routing. Add difficulty routing only after you have data showing many questions are simple and that the small model handles them at your quality bar. Routing errors are hard to see: a wrongly routed hard question gets a weak answer with no error anywhere.

Model version pinningCopy Anchor Link

Providers update models, and behavior changes. A prompt that worked last month can degrade silently.

• Pin exact model versions in production when providers offer them, rather than "latest" aliases.
• Track deprecation dates for every pinned version, and plan migrations months ahead.
• Treat a model upgrade like a code change: run the evaluation set, compare, canary, then roll out. See Rollouts.

Self-hosted modelsCopy Anchor Link

If you self-host open-weight models:

• Use a serving engine built for throughput, such as vLLM, TGI, or TensorRT-LLM. They batch requests continuously and manage GPU memory for long contexts.
• Expose an OpenAI-compatible API from the serving engine, and add it as one more adapter in the gateway.
• Plan capacity for peak concurrent requests and context length, not average. GPU memory for long contexts is usually the limit.
• Monitor GPU utilization. Idle GPUs are the largest waste in self-hosting.

See Infrastructure for hosting details.

ChecklistCopy Anchor Link

• Each task has a chosen model backed by evaluation results.
• Feature code uses logical model names only.
• Routing, fallbacks, and limits are configured in one place.
• Production uses pinned model versions with known deprecation dates.
• Fallback models are evaluated with the same prompts.
• Usage and cost are recorded for every call.
• Tenant data rules (providers, regions, retention) are enforced by the gateway.