Model routing
Model routing: measure the cost of a finished task, not the cheapest token
Compare model routes by cost per accepted task, including retries, fallbacks, tools, and human review. Includes hypothetical arithmetic and a routing worksheet.
Choose a model route by the total cost of producing an accepted result at the quality and speed your workflow needs. Token prices help estimate a model call. They do not tell you what a finished task will cost after retries, escalations, tool calls, and human correction.
A route is the rule that chooses what happens next: deterministic code, a smaller model, a larger model, or a person. The simplest useful route may be a few explicit conditions. You do not need another AI model to decide every AI call.
When routing helps—and when it does not
Routing is worth testing when tasks differ enough to justify different treatments. A short classification request and a difficult research synthesis may not need the same model, context, or review process.
Ollama’s September 29, 2026 announcement adds local decision models for bounded tasks such as triage and routing. That creates an option to evaluate; it does not prove a decision model will make the correct choices for your workload.
Avoid routing complexity when volume is low, task categories are unclear, or you cannot judge the outputs. First establish a straightforward baseline. Also rule out tasks that ordinary parsing or business rules can handle without a model.
Define acceptance before comparing costs
For a synthetic support-routing workflow, an accepted result might mean the correct team, a supported urgency label, and escalation when the input is ambiguous. The record must also reach the intended queue. Valid JSON is not enough.
Use the same labeled inputs and acceptance rules for every candidate. Include routine requests, missing details, conflicting instructions, and examples that should go to a person. Repeat trials where outputs vary and report sample size and failures.
Divide all included batch costs—including failed attempts—by accepted tasks. Report human-review time separately unless its cost conversion is explicit.
Anthropic’s January 9, 2026 evaluation guide emphasizes explicit task criteria and checking outcomes. That is the foundation for a meaningful routing comparison.
Worked example: cheap calls, expensive completion
These invented numbers use arbitrary cost units. They are not provider prices, benchmarks, or measured results. Assume the compared costs include all billable calls, tools, and infrastructure for this batch; human effort is recorded separately.
Consider 100 fictional intake requests. A small-model-first route makes 100 initial calls costing 1 unit each. It escalates 80 requests to another model at 4 units each. Routing and tool overhead adds 80 units. Only 90 final results meet the agreed criteria.
A direct-model baseline costs 5 units per request, with no additional costs assumed in this example, and produces 98 accepted results.
| Hypothetical batch | Initial calls | Fallback calls | Other cost | Total | Accepted | Cost per accepted task |
|---|---|---|---|---|---|---|
| Small-model-first | 100 × 1 | 80 × 4 | 80 | 500 | 90 | 5.56 units |
| Direct baseline | 100 × 5 | 0 | 0 | 500 | 98 | 5.10 units |
The arithmetic is 100 + 320 + 80 = 500; then 500 ÷ 90 ≈ 5.56. The baseline is 100 × 5 = 500; then 500 ÷ 98 ≈ 5.10.
The cheaper initial call did not produce cheaper accepted work. If the hypothetical acceptance floor is 95 results out of 100, the routed batch fails it. A favorable token bill cannot override that requirement. Do not use these invented rates to predict another workload.
Worksheet
Routing worksheet
Fill this before implementing. Scroll the tables horizontally on a narrow screen.
| Decision | Your rule or measurement |
|---|---|
| Task and input types | ___ |
| Deterministic path, if available | ___ |
| Accepted result and quality floor | ___ |
| Initial route and eligible inputs | ___ |
| Escalation trigger and destination | ___ |
| Retry, time, and spend limits | ___ |
| Human-review rule and owner | ___ |
| Data allowed for each provider | ___ |
| Test set, versions, and baseline | ___ |
| Rollback condition | ___ |
For each batch, record total submitted tasks, accepted results, unresolved failures, total cost, escalation rate, and human-review minutes. Measure median and slow-tail completion time from the initial request to acceptance—not just the first model response. Keep performance visible by task category so easy cases do not conceal expensive failures.
Failure mode: a fallback that changes the task
An outage fallback may use different tools, data-handling terms, or output limits. It may also cost more than expected. Treat fallback as a tested route with its own permissions and budget, not an unlimited escape hatch. If no authorized route can satisfy the task, stop and explain the limit.
Cloudflare’s dynamic-routing documentation, updated October 2, 2026, describes conditional routes, quotas, fallbacks, and versioned changes. Those controls manage execution; your outcome checks still have to establish quality.
Acceptance checklist
- Compare routes on the same inputs and acceptance rules.
- Count failed work, retries, fallbacks, and checking costs.
- Record human correction separately unless its cost conversion is explicit.
- Enforce provider permissions, review rules, and limits on every route.
- Test errors and ambiguous cases, not only easy requests.
- Keep a baseline and a tested rollback path.
For a smaller classification step, see when a decision model fits better than a general LLM.
Sources were checked October 7, 2026. The worksheet and arithmetic are illustrative guidance, not promised savings or performance.