Most production traffic doesn't need your most expensive, slowest model. Model routing inspects each request and dispatches it to the cheapest model that can do the job well - using classifiers, cascades, semantic caches, and fallbacks. The payoff is dramatic cost and latency savings without sacrificing quality on the requests that actually need it.
Model routing is the layer between your application and the model providers that decides which model handles each request. Instead of hard-wiring every call to one flagship model, a router assesses the incoming request - its difficulty, cost budget, and latency target - and picks from a fleet of models that trade off price, speed, and capability. The best routers make requests cheaper and faster on average while keeping quality where it matters.
Score the request's difficulty and requirements - length, task type, required reasoning - before choosing a model.
Send simple requests to a small, cheap, fast model and hard ones to a larger, stronger model.
On error, rate-limit, or timeout, retry on another model or provider so a single outage never takes you down.
| Strategy | What it does | Best for |
|---|---|---|
| Classifier / predictive | Assess difficulty up front, then pick a model per request | Mixed traffic with easy + hard queries |
| Cascade / escalation | Try a cheap model, escalate only on low confidence | When most requests are easy but some aren't |
| Semantic cache | Serve a stored answer for semantically similar queries | Repetitive, high-volume, FAQ-style traffic |
| Fallback routing | Switch provider on error, rate-limit, or timeout | Reliability and multi-provider resilience |
| Rule / metadata | Route by tenant, region, tier, or data-sensitivity | Compliance, SLAs, and cost tiers |
Flagship models can cost 10–30× more per token and run several times slower than small ones. When every request hits the biggest model, you overpay massively for the easy 80% of traffic - and users wait longer than they need to. Routing recovers that waste.
Serving simple queries from a small model or a cache - instead of a flagship - can slash spend by large multiples on high-volume traffic.
Small models and cache hits return in a fraction of the time. Users get faster answers on the requests that don't need deep reasoning.
Hard requests still reach the strong model. You protect quality on the queries that matter instead of flattening everything to one tier.
Multi-provider fallback means a single provider's outage, rate-limit, or timeout doesn't break your product - traffic reroutes automatically.
Same idea, four ways to picture it, so it clicks whoever you are.
Routing is like knowing when to use a calculator versus scratch paper. Easy arithmetic you do in your head; only the hard problems get the heavy tool. You match the effort to the question instead of reaching for the biggest tool every time.
It's a load balancer for intelligence. Just as you route cheap reads to a replica and heavy writes to the primary, the router sends easy prompts to a small model and hard ones to the flagship, with a fallback pool when one backend is down.
Think of a manager delegating work. Routine tasks go to a junior who is fast and inexpensive; only the tricky, high-stakes cases escalate to the senior expert. You reserve the pricey specialist for where it truly pays off.
A triage nurse at the ER sends each patient to the right level of care. Minor cuts go to a quick station; serious cases go straight to the specialist. Nobody wastes the surgeon's time on a scraped knee.
A router sits in front of the model providers and juggles three concerns at once - the cost / latency / quality triangle. You can optimize any two, but tightening one usually pressures the others, so routing is about landing each request at the right point on that triangle.
Cost, latency, and quality pull against each other. A cheap fast answer may be lower quality; a top-quality answer costs more and takes longer. Routing places each request where its SLA and budget allow.
Log cost, p50/p95 latency, and a quality score per route. Set a latency SLA and a per-request budget, then let the router optimize within those guardrails and alert when they're breached.
flowchart LR
Q["❓ Request"] --> R{"🧭 Router"}
subgraph Fast["⚡ Fast paths"]
C[("🧊 Semantic cache")]
SM["🐣 Small model"]
end
subgraph Strong["🧠 Strong paths"]
LG["🦾 Large model"]
FB["🛟 Fallback provider"]
end
R --> C
C -->|hit| ANS["✅ Response"]
C -->|miss| CL["📊 Classify difficulty"]
CL -->|easy| SM
CL -->|hard| LG
SM -->|low confidence| LG
LG -->|error / timeout| FB
SM --> ANS
LG --> ANS
FB --> ANS
def route(request):
# 1. Semantic cache: serve similar past answers
hit = cache.lookup(embed(request), threshold=0.92)
if hit:
return hit.answer
# 2. Predictive routing: assess difficulty, pick a tier
score = classifier.difficulty(request) # 0.0 .. 1.0
model = small_model if score < 0.5 else large_model
# 3. Cascade: try cheap first, escalate on low confidence
resp = call(model, request)
if model is small_model and resp.confidence < 0.7:
resp = call(large_model, request) # escalate
# 4. Fallback: on error / rate-limit / timeout, switch provider
if resp.error:
resp = call(fallback_provider, request)
cache.store(embed(request), resp.answer)
return resp.answerNotice the ordering: cheapest checks first (cache), then a light classifier, then the cascade, with fallback wrapping everything. Each layer only pays for the next when it has to - that's what keeps average cost and latency low.
Three views: a classifier dispatching to the right model, a cheap-first cascade with confidence-based escalation, and a semantic cache with provider fallback on failure.
sequenceDiagram
autonumber
participant U as 👤 User
participant R as 🧭 Router
participant C as 📊 Classifier
participant S as 🐣 Small model
participant L as 🧠 Large model
U->>R: Send request
R->>C: Score difficulty
C-->>R: score plus task type
alt easy request
R->>S: Dispatch to small model
S-->>R: Fast cheap answer
else hard request
R->>L: Dispatch to large model
L-->>R: High quality answer
end
R-->>U: Response plus route metadata
sequenceDiagram
autonumber
participant U as 👤 User
participant R as 🧭 Router
participant S as 🐣 Small model
participant J as 🔍 Confidence check
participant L as 🧠 Large model
U->>R: Send request
R->>S: Try cheap model first
S-->>R: Draft answer plus signal
R->>J: Evaluate confidence
J-->>R: score
alt confident enough
R-->>U: Return cheap answer
else low confidence
R->>L: Escalate to strong model
L-->>R: Higher quality answer
R-->>U: Return escalated answer
end
Note over R,L: Only hard requests pay for the big model
sequenceDiagram
autonumber
participant U as 👤 User
participant R as 🧭 Router
participant C as 🧊 Semantic cache
participant P as 🟣 Primary provider
participant F as 🛟 Fallback provider
U->>R: Send request
R->>C: Lookup similar query
alt cache hit
C-->>R: Stored answer
R-->>U: Instant cached response
else cache miss
C-->>R: No match
R->>P: Call primary provider
alt provider healthy
P-->>R: Answer
else error or rate limit or timeout
P-->>R: Failure
R->>F: Retry on fallback
F-->>R: Answer
end
R->>C: Store answer for reuse
R-->>U: Fresh response
end
From a raw request to a delivered response - every decision the router makes, in order.
A request arrives at the routing layer with its prompt, tenant, and any SLA or budget hints attached as metadata.
Normalize the prompt and compute an embedding so it can be compared against the semantic cache and scored for difficulty.
Look up semantically similar past queries. If one is close enough above the similarity threshold, serve its stored answer and skip the model entirely.
On a cache miss, a lightweight classifier scores the request - task type, reasoning depth, length - to predict which model tier can handle it.
Combine the difficulty score with the request's latency SLA and cost budget to choose a candidate model within the allowed envelope.
Send easy requests to a small, cheap, fast model and hard ones straight to a stronger model. Metadata records the route taken.
For cascade routes, inspect the cheap model's confidence - self-reported score, logprobs, or a cheap judge - to decide if the answer is good enough.
If confidence is below the threshold, escalate to a larger model. Only the genuinely hard requests pay the premium price and latency.
On error, rate-limit, or timeout, retry on an alternate model or provider with backoff so a single outage never surfaces to the user.
Write the final answer back to the semantic cache so the next similar query is served instantly and cheaply.
Return the response with route metadata, and log cost, latency, and quality per route so dashboards and evals can tune thresholds over time.
Routing saves money right up until it quietly breaks quality or reliability. These are the usual culprits.
A router that underestimates difficulty sends genuinely hard requests to the small model and never escalates. The result looks cheap on the dashboard but underserves exactly the users who needed the strong model most.
Too many escalation hops mean a request runs the small model, fails a confidence check, then reruns the large model - paying for both and doubling latency. Deep cascades can be slower and costlier than going big up front.
A semantic cache with a loose similarity threshold returns a stored answer for a subtly different question, or serves outdated content after the source changed. Wrong-but-fast is worse than slow-but-right.
Routing to a single provider with no alternate means one outage, rate-limit, or timeout takes the whole product down. Cost optimization is meaningless if the service isn't available.
Optimizing purely for cost, with no quality signal in the loop, degrades answers so gradually nobody notices until users churn. Cheap traffic that quietly regresses is a hidden tax on trust.
A heavy classifier, embedding call, or cache lookup on every request adds its own latency and cost. If the routing overhead rivals the savings, the layer is defeating its own purpose.
Track cost and latency alongside quality, so savings never come at the expense of the answers.
| Metric | What it tells you | Good sign |
|---|---|---|
| Cost per request | Average spend per call across the whole route mix | Low: routing is capturing the savings |
| p50 / p95 latency | Typical and tail response time, including routing overhead | Within your SLA at both percentiles |
| Quality / win-rate vs always-big-model | How often routed answers match or beat sending everything to the flagship | High: little quality lost for the savings |
| Escalation rate | Share of requests the cascade promotes to a stronger model | Stable and moderate: not too eager, not starving hard queries |
| Cache hit rate | Fraction of requests served from the semantic cache | High without wrong-answer complaints |
| Fallback / error rate | How often the primary path fails and reroutes | Low, with fallbacks catching what does fail |
Three representative patterns showing routing in production-style use.
A consumer chatbot handles millions of messages a day, most of them simple greetings, FAQs, and short follow-ups.
A data team must label millions of documents overnight within a fixed batch window.
A business-critical product cannot afford downtime when a single model provider degrades.
Routing is evolving from static rules into learned, adaptive systems that optimize cost, latency, and quality in real time.
Models trained on real traffic that predict the cheapest model likely to satisfy each request - replacing hand-tuned rules with data-driven policies.
Routers that optimize against an explicit cost ceiling and latency SLA per request, spending the budget only where quality demands it.
Caches that reason about freshness and partial reuse - serving cached fragments and refreshing only what has changed.
Unified gateways that treat many providers as one fleet, load-balancing and failing over transparently across vendors.
Online quality scoring per route so the system detects drift and re-tunes thresholds automatically instead of on a manual cadence.
Small models drafting for large ones, blurring the line between routing and generation to cut latency without losing quality.