LLM routing

Overview

Language models differ enormously in price, speed and ability. The most capable models can cost many times more per token than small ones and take longer to answer. Yet most real traffic is a mix: many requests are simple, such as a greeting, a reformatting job or a factual lookup, and a minority are genuinely hard. Sending everything to the most expensive model wastes money; sending everything to the cheapest model hurts quality on the hard cases.

LLM routing solves this by deciding, request by request, which model should answer. The decision can be as simple as a rule ("code questions go to the coding model") or as sophisticated as a trained router that predicts whether a small model's answer will be good enough. A related pattern, the cascade, tries a cheap model first and escalates to a stronger one only if the first answer looks unreliable. Routing also covers reliability: if one provider is down or rate-limited, a fallback sends the request elsewhere.

An everyday analogy: a hospital triage nurse does not send every patient to the senior consultant. A sprained wrist goes to a nurse practitioner, chest pain goes straight to the cardiologist, and anything unclear is examined quickly first and escalated if needed. Patients get the right level of care, and the consultant's time is spent where it matters. An LLM router is the triage desk for requests.

What it is

Routers fall into three broad types. Rule-based routers use hand-written conditions such as keywords, request metadata or prompt length. Classifier-based routers use a trained model, often a small encoder, to label each request by task or difficulty and map labels to models. Learned routers go further and directly predict which model will give the best result, or whether a cheap model's answer will be acceptable, usually by learning from preference data or past outcomes.

It helps to separate routing from mixture of experts. A mixture-of-experts model routes tokens between sub-networks inside a single model, and that routing is learned during training. LLM routing chooses between whole, separate models at the application level, before any model runs.

Where it came from

Routing between models of different cost is an old idea in machine learning. Cascades of classifiers, where cheap stages reject easy cases and expensive stages handle the rest, were used for decades in tasks such as face detection. For large language models the idea gained momentum in 2023, as many models with very different prices became available through APIs.

In May 2023, Lingjiao Chen, Matei Zaharia and James Zou at Stanford published "FrugalGPT", which grouped cost-saving strategies into prompt adaptation, LLM approximation and LLM cascades. Their cascade learned which models to query for each input and reported matching the best single model's performance at up to 98 per cent lower cost in their experiments. In March 2024, Qitian Jason Hu and colleagues released RouterBench, a benchmark of over 405,000 inference outcomes for evaluating routing systems. At ICLR 2024, Dujian Ding and colleagues presented "Hybrid LLM", a router that chooses between a small and a large model based on predicted query difficulty, with a quality level that can be tuned at test time; they reported up to 40 per cent fewer calls to the large model with no drop in response quality.

In June 2024, Isaac Ong and colleagues, including researchers from UC Berkeley and Anyscale, published RouteLLM, later presented at ICLR 2025. It trained routers on human preference data from Chatbot Arena to choose between a strong model and a weak one, compared four router designs, and reported cost reductions of over two times in some settings without losing quality. The team also released an open-source framework.

Routing then moved into products. OpenAI's GPT-5 system card, published in August 2025, describes a real-time router in ChatGPT that chooses between a fast model and a deeper reasoning model based on conversation type, complexity, tool needs and explicit user intent, and that is trained on signals such as users switching models, preference rates and measured correctness. Gateways such as LiteLLM and OpenRouter offer routing, load balancing and fallbacks as standard features.

How it works

Rule-based routing

The simplest router is a set of conditions. Requests from free users go to a small model; requests with attachments go to a multimodal model; requests longer than a context limit go to a long-context model; requests flagged as code go to a coding model. Rules are transparent, cheap and easy to audit, but they cannot judge how hard a particular question is.

Classifier-based routing

A small classifier, often a fine-tuned encoder that takes embeddings of the request, assigns a category or a difficulty score. Each category maps to a model. This handles fuzzy cases that keywords miss, adds only milliseconds of latency, and can be retrained as traffic changes. Its weakness is that categories are a proxy: two "maths" questions can differ greatly in difficulty.

Learned routers

Learned routers predict quality directly. RouteLLM's designs illustrate the range: a similarity-weighted ranking method that needs no training and looks up similar past comparisons; a matrix factorisation model that scores how well each model suits a query; a BERT classifier; and a Llama 3 8B model fine-tuned to predict which model would win. All of them output a probability that the strong model is needed, and a threshold turns that into a decision.

The decision rule

Router score:     s(x) = P( strong model needed | x )
Decision:         send x to the strong model  if  s(x) ≥ τ,  otherwise to the weak model
Expected cost:    E[C] = f(τ) · c_strong + (1 − f(τ)) · c_weak
Cascade:          answer with m₁; if q(x, y₁) < θ, escalate to m₂

Here x is the request, s(x) is the router's estimate that the cheaper model will fall short, and τ is the threshold you choose. f(τ) is the fraction of traffic sent to the strong model at that threshold, and c_strong and c_weak are the average costs per request of each model, so E[C] is the expected cost. In the cascade line, m₁ and m₂ are the cheap and strong models, y₁ is the cheap model's answer, q is a quality or confidence check on that answer, and θ is the acceptance threshold. Raising τ or lowering θ saves money; doing the opposite buys quality.

Cascades and fallbacks

A cascade runs the cheap model first and checks the answer: by self-reported confidence, token probabilities, a verifier model, a format check or a test. If the answer passes, it is returned; if not, the request goes to a stronger model. Cascades guarantee that hard cases get a strong model but add latency and cost for escalated requests, because they pay for two calls.

Fallbacks address reliability rather than quality. If a provider returns an error, times out or hits a rate limit, the gateway retries or switches to an equivalent model elsewhere. LiteLLM, for example, documents several routing strategies (including simple shuffle, least busy, latency-based and cost-based), retries, cooldowns for failing deployments, and separate fallbacks for general failures, context-window overflows and content-policy refusals. These patterns resemble the circuit breaker used in distributed systems.

A code example

The plain-Python sketch below shows the logic of a two-stage cascade with a confidence check. The call_model and confidence functions stand for whatever client and check you use.

def route(prompt, call_model, confidence, threshold=0.8):
    """Try the cheap model first, escalate if the answer looks unreliable."""
    cheap_answer = call_model("small-model", prompt)
    if confidence(prompt, cheap_answer) >= threshold:
        return cheap_answer, "small-model"
    strong_answer = call_model("large-model", prompt)
    return strong_answer, "large-model"

In production, the same function would also log the choice, the cost and the latency so that the threshold can be tuned against real outcomes.

Key concepts and terms

A worked example

Consider an illustrative customer support assistant handling 100,000 requests a day. A large model gives excellent answers but costs ten units per request; a small model costs one unit. Sending everything to the large model costs 1,000,000 units a day.

The team builds an evaluation set of 2,000 real, anonymised requests and has both models answer each one. Using an LLM-as-judge with human spot checks, they find the small model's answer is acceptable on about 70 per cent of requests. A perfect router would send only the remaining 30 per cent to the large model, for a daily cost of 70,000 + 300,000 = 370,000 units.

Their router is not perfect. They train a small classifier on the evaluation labels and plot a deferral curve: quality against the share of traffic sent to the large model. At a threshold that sends 40 per cent of traffic to the large model, quality on the held-out set is within one percentage point of the all-large baseline. Daily cost becomes 60,000 + 400,000 = 460,000 units, a 54 per cent saving. They add rules on top: anything mentioning refunds or legal terms always goes to the large model, and if the large model's provider fails, requests fall back to a second provider.

After launch they watch the logs. A month later, a new product line appears and the small model starts failing on those questions. The router, trained on old traffic, keeps sending them to the small model. The team adds the new questions to the evaluation set, retrains the router, and puts a monthly retraining step in their process. These numbers are illustrative, but the pattern of evaluate, route, monitor and retrain is the real work of routing.

Strengths and limitations

Strengths:

• Large cost savings when much of the traffic is easy, while keeping strong models for hard cases.
• Lower average latency, since small models answer faster.
• Better reliability through fallbacks across models and providers.
• Flexibility to add new models without changing application code, especially behind a gateway.
• A tunable dial between cost and quality that can differ by user tier or feature.

Limitations:

• Routers make mistakes; sending a hard question to a weak model produces a visibly worse answer.
• Learned routers drift when traffic, models or prices change, and need retraining.
• Cascades add latency and double cost on escalated requests.
• Different models behave differently in tone, format and safety, which can make the product feel inconsistent.
• Evaluation is costly: you need answers from every candidate model on representative data.
• Routing adds another component to monitor, debug and secure.

How it compares with alternatives

Routing pairs well with the others. A common design routes easy traffic to a distilled small model and hard traffic to a reasoning model with a generous thinking budget, which combines routing with test-time compute.

Practical guidance and best practices

1. Start with an evaluation set drawn from real traffic, and collect answers from every candidate model on it.
2. Begin with rules for clear-cut cases, such as safety-critical topics or modality, and add a learned router only where rules cannot judge difficulty.
3. Plot a deferral curve and choose the threshold from data, not intuition.
4. Keep the router fast and cheap; a router that costs as much as the small model defeats the purpose.
5. Log every decision with cost, latency and, where possible, an outcome signal such as user feedback or task success.
6. Retrain or recalibrate the router when models, prices or traffic change, and re-run the evals after every model swap.
7. Put routing, retries and fallbacks in one gateway layer rather than scattering them through application code.
8. Keep output format consistent across models with shared instructions and structured outputs, and apply the same guardrails to every route.
9. Use tracing so that a bad answer can be traced back to the model and the routing decision that produced it.

Common mistakes

• Optimising the router for cost alone and discovering quality problems only through user complaints.
• Evaluating the router on synthetic or benchmark questions that do not resemble real traffic.
• Forgetting that cascades double the cost and latency of escalated requests.
• Letting a learned router go stale after a model upgrade or a price change.
• Ignoring differences in tone, refusal behaviour and formatting between models.
• Building fallbacks without testing them, so the backup path fails when it is finally needed.

Frequently asked questions

What is the difference between a router and a cascade?

A router decides up front which model to call, based on the request alone. A cascade calls a cheap model first, checks its answer, and escalates only if the answer fails the check. Routers are faster; cascades can be more accurate on borderline cases because they see an actual answer before deciding.

How much can routing save?

It depends entirely on your traffic and models. Research papers report large savings in their settings, such as RouteLLM's over two times cost reduction in some cases and FrugalGPT's up to 98 per cent, but those numbers come from specific benchmarks. Measure on your own data before promising any figure.

Do I need a machine learning model to route?

No. Many production systems start with rules based on task type, user tier, prompt length or modality. A learned router becomes worthwhile when you need to judge difficulty within the same kind of request and have enough labelled data to train it.

How is LLM routing different from mixture of experts?

Mixture of experts is an architecture inside a single model, where a learned gate sends each token to a few expert sub-networks. LLM routing chooses between whole, independent models at the application level, and the models can come from different providers.

How do I evaluate a router?

Collect answers from every candidate model on a representative set, score them, and measure how much quality the router keeps as you vary the share of traffic sent to the expensive model. Metrics such as performance gap recovered and deferral curves summarise this trade-off.

Should routing live in my application or in a gateway?

A gateway is usually better once you have more than a couple of models. It centralises routing rules, retries, fallbacks, rate limits, logging and cost tracking, and lets you change models without redeploying the application.

Glossary

• LLM routing: choosing a model for each request.
• Router: the component that makes that choice.
• Cascade: escalation from a cheap model to a stronger one after a failed check.
• Fallback: a backup route used on errors or outages.
• Gateway: a single API layer in front of many models.
• Threshold: the cut-off that converts a router score into a decision.
• Deferral curve: quality against the share of traffic sent to the strong model.
• Performance gap recovered: the fraction of the weak-to-strong quality gap a router closes.
• Cooldown: temporary removal of a failing deployment.
• Drift: change in traffic or models that makes a router's past learning out of date.

Summary

LLM routing sends each request to the model that best balances quality, cost and speed for that request. It ranges from simple rules through classifiers to learned routers trained on preference data, and it includes cascades that escalate only when a cheap answer fails a check and fallbacks that keep the system running during outages. Research such as FrugalGPT, Hybrid LLM, RouterBench and RouteLLM showed that large savings are possible, and products now ship routers and gateways as standard. The value depends on your traffic, so the real work is building a representative evaluation set, choosing thresholds from deferral curves, logging every decision and retraining whenever models, prices or traffic change.

Related

It belongs to Frontier topics in my stack, under Generative AI and LLMs. Others in the same group: