One request. Multiple models. One intelligent route.
Every request is routed to a model chosen on cost, latency, quality and data sensitivity — with a fallback chain behind it. A simple interface; a real decision layer underneath.
- Local + cloud inference
- Per-request cost tracking
- Automatic failover
Your prompt
“Summarise this contract and flag the risks”
Router
Classify → score → decide → dispatch
- Cloud — large contextBest for long analysis
- Cloud — fast tierCheaper, shorter context
- Local Llama 3Free, private, slower
- Cloud — reasoningOverkill for this task
Chosen for long-context analysis · est. $0.011 · local fallback ready
Why not send everything to one model?
Because “best model” is not a property of a model. It's a property of a request.
Cost
A classification task does not need your most expensive model. Sending it there anyway is the single largest line item on most teams’ first LLM invoice.
Latency
An autocomplete has a budget measured in hundreds of milliseconds. A nightly summarisation job does not. The same model cannot be right for both.
Quality
Some requests genuinely need the strongest available model. Knowing which ones is the decision; sending everything there is the absence of one.
Capability
Context window, tool use, structured output, vision. A request that needs 200k tokens of context has a shortlist of one or two, not ten.
Data sensitivity
Some payloads must never leave your network. That is not a preference to weigh against cost — it is a hard constraint that removes candidates outright.
Availability
Providers have outages. A system with one provider and no fallback chain has an availability ceiling it does not control.
See the decision, not just the answer
Type a prompt and the demo shows you what it classified the request as, which models were candidates, what each would have cost, which one it picked and why. The dashboard then aggregates that across every request the system has served.
System architecture
The provider integrations are the easy half. The policy layer is the system.
Client
Router API
Policy layer
Providers
Observability
Key engineering decisions
- Why a policy layer instead of a rule in each caller?
- Because routing rules change far more often than the code that makes requests. With the decision in one layer, changing "classification goes to the cheap tier" is one deployment, not a search across every service that calls a model. It is also the only place a per-request cost estimate can exist before the request is sent.
- Why does the provider abstraction come before the providers?
- Adding a provider should be implementing one interface, not editing the router. The interface was written first and the providers implemented against it, which is why local Ollama and a cloud API are interchangeable candidates rather than two code paths with an if-statement between them.
- Why plain SQL for the dashboard rather than an ORM?
- The dashboard’s queries are aggregates — percentile latency, filtered counts, cost sums. An ORM would either fight those or hide them behind a raw-query escape hatch, at which point it is a dependency adding nothing. Plain pg keeps the queries readable by anyone who knows SQL.
- What happens when the selected provider fails?
- The decision carries a fallback chain, not just a winner. A failed call moves to the next eligible candidate and records both the failure and the substitution, so the dashboard shows you which provider is costing you reliability — not just which is costing you money.
What I'd build next
What separates this demo from the production version of it.
Adaptive routing
Feed measured quality back into the scoring so the router learns which model actually performs on each request class, rather than trusting a static table.
Budget-aware routing
A spend ceiling per tenant per period, with graceful degradation to cheaper or local models as the budget depletes instead of a hard cut-off.
Latency-aware routing
Route on observed p95 per provider in the last few minutes, not on published figures — providers slow down long before they fail.
Evaluation feedback
An offline eval set per request class, re-run on every catalogue change, so adding a model is a measured decision rather than a hopeful one.
Provider failover policy
Circuit breakers per provider with half-open probes, so one degraded provider stops absorbing retries from every request in flight.
Semantic caching
Near-duplicate prompts served from cache with a similarity threshold — the cheapest request is the one never sent.
An AI workload that needs routing, reliability or cost control?
Most teams reach this problem about six months after their first LLM feature ships, when the invoice arrives and one provider has an outage in the same week. I've designed the decision layer that answers both. I'd be glad to design yours.