Must a model router use machine learning?

No. Rules can be sufficient when task boundaries and requirements are clear.

Does routing always save money?

No. Selection overhead, retries, model prices and workload composition can outweigh the savings.

How is a routing threshold chosen?

Evaluate candidate thresholds on representative held-out requests against explicit quality, latency and cost requirements, then monitor whether those relationships persist.

AI Glossary / Statistics and probability / Intermediate

Model router

Definition

A model router is a component that chooses which available model should handle a particular input or task. Its decision can depend on predicted answer quality, cost, response time, capability, availability or other requirements. A router may be a learned model, a set of explicit rules or a combination. In language-model systems, routing can direct simpler requests to a smaller model and more demanding requests to a stronger or specialized model. The choice is an estimate under an objective, not a guarantee that the selected model will give the best answer.

What it means

Different requests can benefit from different models. A short classification task may not require the same resources as a complex reasoning task, and a specialist may suit a domain better than a general model. Routing seeks to exploit those differences. The router itself consumes resources and can make mistakes. Its value therefore depends on the complete service, including selection overhead, model calls, retries and final response quality. Choosing the cheapest model every time is a routing policy, but it does not necessarily satisfy the service's quality requirements.

How it works

Define the eligible models, the permitted destinations and the quality and resource objectives. Build a routing policy using request characteristics, historical evaluations or explicit capability rules. If the router is learned, train it on representative examples with suitable labels or preference information. Calibrate its decision thresholds on held-out data and evaluate the resulting end-to-end system. Record the selected model, relevant versions and outcomes. Include handling for unavailable models or uncertain decisions, then monitor changes in traffic, model behavior and the relationship between routing scores and actual quality.

Three practical examples

Customer assistance: send straightforward classification requests to a compact model while reserving complex synthesis for another model.
Multilingual processing: choose a model whose demonstrated capabilities match the request's language and task.
Internal automation: route requests subject to particular data-handling requirements only to eligible approved destinations. In every case, compare the quality of final outcomes rather than judging success solely by whether the routing rule executed.

Think of it this way

Think of a reception desk assigning requests to appropriate specialists. A good assignment considers the problem, available expertise and service constraints. The desk also takes time and can misunderstand a request. A model router has the same practical limitation: even excellent specialists can produce poor overall service if requests reach the wrong destination.

Worked example

Consider 1,000 artificial requests. Model S costs 1 credit per request and model L costs 8. A router costs 0.1 credit per request and sends 750 requests to S and 250 to L. Total cost is 750 times 1 + 250 times 8 + 1,000 times 0.1 = 2,850 credits. Sending every request directly to L costs 8,000, so the measured cost reduction is 64.375%. Suppose held-out evaluation finds 720 correct responses in the S group and 240 in the L group. Overall accuracy is 960/1,000 = 96%. If routing errors instead leave only 700 correct in the S group, accuracy drops to 94% despite identical cost. This original example assumes fixed costs and a binary correctness criterion. It does not establish equality with L's quality, which must be measured separately on the same workload.

Illustrative case study

In a fictional case, Birch Services tests a router for an internal writing assistant. Its initial evaluation shows lower average cost, but the team discovers that requests with short wording can still require difficult reasoning. It adds task-specific evaluation cases and checks the downstream result instead of relying on prompt length alone. The deployment records model choices and includes a fallback for uncertain requests. Managers approve expansion only after examining quality by task type and accounting for the router's own processing cost.

Common misconceptions and mistakes

Routing is not automatically an ensemble that combines several outputs. A router can choose one model before generation, while a cascade may invoke another model after inspecting an earlier result. Routing among complete models also differs from routing tokens among internal experts in a mixture-of-experts architecture, although the ideas are related. A stronger model is not best for every request. A well-calibrated threshold on yesterday's traffic can become unsuitable after a workload shift. Model unavailability, retries and variable output lengths can change the economics substantially.

Business use

Businesses should compare routing against clear baselines such as one consistently selected model and a simple rule-based policy. Measure task quality, cost, latency, failure rates and performance for important user groups. Check whether the router sends sensitive or restricted requests only to permitted destinations. Retain enough traceability to investigate an incorrect routing decision without collecting unnecessary content. Treat policy and model-version changes as changes to the service itself. The business case should account for evaluation and maintenance effort as well as reduced inference spending.

Check your understanding

A router reduces inference cost by half. Does that demonstrate a successful deployment?

Answer: No. The evaluation must also show that the resulting service meets its quality and operational requirements, including the router's own overhead.

Related terms

Model cascading
Inference
Mixture of experts
Model selection
Latency
quality evaluation
fallback
cost optimization

Learn next

Learn to evaluate complete inference pipelines, then study threshold calibration and workload drift. Compare a learned router with a simple policy using the same requests, scoring rules and accounting boundaries.

Inference

Sources

Google Machine Learning Glossary, model router: https://developers.google.com/machine-learning/glossary/#model-router Ong and colleagues, RouteLLM: Learning to Route LLMs with Preference Data: https://arxiv.org/abs/2406.18665 RouteLLM author repository, routing thresholds and evaluation: https://github.com/lm-sys/RouteLLM

Previous: Model risk management Next: Model selection