A Buyer's Guide to Model routing and fallback

Fallback is designed as an availability control: if the primary model is unreachable, route to a secondary and keep serving. That works for infrastructure, where the replacement does the same thing. It does not transfer cleanly here, because the secondary model is a different model. It formats differently, refuses different things, and interprets the same prompt differently, so during a failover your users get answers from a system nobody evaluated for this task, and nothing in the response indicates that.

Infrastructure failover replaces a component with an identical one. Model failover replaces it with something that behaves differently.

Model routing and fallback means directing requests across models with the behavioural difference understood and surfaced, rather than treating a secondary as a drop-in replacement.

Why Great CTOs Don't Just Build, They Evaluate

Learn how disciplined evaluation separates credible AI systems from hype.

Download Whitepaper

However, most designs test that failover works, meaning that requests succeed, and never test what the answers look like when it does.

If you are a CTO or Head of AI at an enterprise, the intent of this article is:

Define why model failover is not availability failover
Show what has to be evaluated on the secondary
Lay out when failing closed is the right answer

To do that, let's start with the basics.

What Is Model Routing and Fallback? The Basic Definition

At a high level, routing sends requests to different models based on cost, capability, or availability, and fallback covers the case where the intended model is unavailable. The distinguishing property is that models are not interchangeable in the way servers are. A secondary will produce differently formatted output, refuse a different set of requests, handle edge cases differently, and respond to the same prompt with different reasoning. Serving from it without evaluation means serving from an unvalidated system, and the degradation is silent.

To compare:

Model fallback designed as availability failover is substituting a different chef when the first is unavailable. Service continues. The dishes are not the ones on the menu, and nobody told the diners.

Why Does Model Routing and Fallback Matter?

Issues that it addresses or resolves:

Outages taking a capability down entirely
Cost optimisation across capability tiers
Vendor dependence on a single provider

Resolved Issues by Routing Done Well

Secondary evaluated for the task before it is needed
Behavioural difference surfaced rather than silent
Fail-closed applied where degraded answers are worse than none

Core Components of Model Routing and Fallback

Secondary evaluation on the same task and prompts
Prompt portability assessed and adjusted
Degradation signalling to consumers
Fail-closed policy for high-consequence paths
Routing decisions logged

Modern Routing Practice

Evaluation suites run against every routing target
Prompts adapted per model rather than assumed portable
Responses marked when served from a fallback
Fail-closed configured per path by consequence
Routing recorded per request for later investigation

These practices prevent silent degradation. Running the evaluation suite against the secondary is what turns a fallback from a hope into a tier.

Other Core Issues They Will Solve

Quality differences known in advance
Investigations able to account for which model answered
High-consequence paths refusing rather than degrading

In Summary: Model fallback substitutes a different behaviour, not an identical component, so the secondary needs evaluating and the degradation needs surfacing.

Importance of Model Routing and Fallback in 2026

Multi-model estates are normal and failover is assumed safe. Four reasons explain why this matters now.

1. Models differ behaviourally.

Formatting, refusal, and reasoning all vary between providers and versions.

2. Prompts are not portable.

A prompt tuned for one model performs differently on another.

3. Degradation is invisible.

A fallback response looks like a primary response.

4. Some paths should stop instead.

A degraded answer in a consequential path can be worse than an error.

Traditional vs. Modern Fallback Design

Availability tested vs. behaviour evaluated
Prompts assumed portable vs. adapted per model
Degradation silent vs. surfaced
Always fail open vs. fail closed by consequence

In summary: A modern design treats the secondary as a different system that has been evaluated.

Details About the Core Components of Model Routing and Fallback: What Are You Designing?

Let's go through each component.

1. Evaluation Layer

Knowing the secondary.

Evaluation decisions:

Suite run against every target
Same task and prompts used
Quality delta documented

2. Portability Layer

Prompts across models.

Portability decisions:

Prompt variants per model maintained
Format differences handled
Refusal differences mapped

3. Signalling Layer

Surfacing degradation.

Signalling decisions:

Responses marked when from fallback
Consumers able to act on the signal
Internal logging of routing

4. Policy Layer

Open or closed.

Policy decisions:

Fail-closed for high-consequence paths
Fail-open where degraded is better than nothing
Policy set per path

5. Observability Layer

Knowing what answered.

Observability decisions:

Routing recorded per request
Quality tracked per target
Investigations able to attribute

Benefits Gained from Routing Done Well

Quality differences known before the outage
Degradation visible to consumers
Consequential paths refusing rather than guessing

How It All Works Together

The team runs its evaluation suite against every routing target using the same task and prompts, which produces a documented quality delta rather than an assumption of equivalence. Prompt variants are maintained per model, because a prompt tuned for one provider's formatting and instruction-following behaves differently on another, and refusal differences are mapped so a path that works on the primary is not blocked on the secondary. Responses served from a fallback are marked so consumers can act on the signal, and routing is logged per request so an investigation can attribute an odd answer to the model that produced it. Fail-closed is configured per path by consequence, since a degraded answer in a consequential workflow is worse than an error that stops it.

Common Misconception

We have fallback configured, so an outage is covered.

Availability is covered and quality is not. The secondary is a different model: it formats differently, refuses a different set of requests, and interprets your prompt differently, so during the failover your users receive answers from a system nobody evaluated for this task. Those answers look exactly like normal ones, so the degradation is invisible to users and to you, and an investigation weeks later has no record of which model produced the odd output. Fallback is a real capability and it requires evaluating the secondary before you need it.

Key Takeaway: Fallback covers availability. The secondary is a different model, so quality is covered only if you evaluated it.

Real-World Routing Design in Action

Let's take a look at how it operates with a real-world example.

We worked with a team whose failover served unevaluated answers, with these constraints:

Run the evaluation suite against every routing target
Maintain prompt variants per model
Mark responses served from fallback

Step 1: Evaluate the Secondary

Same task, same prompts.

Suite run against targets
Quality delta documented
Differences understood

Step 2: Adapt the Prompts

Not portable by default.

Variants per model maintained
Format differences handled
Refusal differences mapped

Step 3: Surface the Degradation

Do not hide it.

Responses marked
Consumers able to act
Routing logged

Step 4: Set the Policy Per Path

Open or closed.

Fail-closed where consequential
Fail-open where useful
Policy per path

Step 5: Record What Answered

For investigation.

Routing per request logged
Quality tracked per target
Attribution possible

Where It Works Well

Estates with evaluation suites that can be rerun
Paths where degraded service is genuinely better than none
Consumers able to act on a degradation signal

Where It Does Not Work Well

Fallback tested only for successful responses
Prompts assumed portable across providers
High-consequence paths failing open silently

Key Takeaway: Evaluate the secondary, adapt the prompts, surface degradation, set policy per path, record what answered.

Common Pitfalls

i) Testing availability only

Confirming that requests succeed on the secondary says nothing about what the answers look like. Run the evaluation suite.

Failover worked
The answers were from an unevaluated system
Nothing indicated it

ii) Assuming prompt portability

A prompt tuned for one model behaves differently on another, sometimes substantially. Maintain variants and map refusal differences.

iii) Silent degradation

A fallback response is indistinguishable from a normal one, so neither users nor operators know. Mark it and log the routing.

iv) Always failing open

In consequential paths a degraded answer is worse than an error. Set the policy per path by consequence.

Takeaway from these lessons: The replacement is not equivalent, which is what makes this different from infrastructure failover.

Routing and Fallback Best Practices: What High-Performing Teams Do Differently

1. Run your evaluation suite against every routing target

Document the quality delta before the outage rather than discovering it during one.

2. Maintain prompt variants per model

Treat prompts as model-specific rather than portable, including refusal behaviour differences.

3. Mark responses served from a fallback

Let consumers and operators know the answer came from a secondary.

4. Set fail-open or fail-closed per path by consequence

Recognise that some paths are better stopped than served degraded.

5. Log routing per request

Make it possible to attribute an odd answer to the model that produced it.

Logiciel's value add is helping teams treat fallback models as different systems requiring evaluation, so failover does not quietly change the answers.

Takeaway for High-Performing Teams: Evaluate targets, adapt prompts, mark fallbacks, set policy per path, log routing.

Signals You Are Doing This Well

How do you know it is working? Not by failover success, but by whether you know the secondary's quality. These are the signals that separate evaluated routing from configured failover.

Targets are evaluated. Quality deltas are documented per model.

Prompts are model-specific. Variants exist and refusal differences are mapped.

Degradation is visible. Fallback responses are marked.

Policy varies by path. Consequential paths fail closed.

Routing is logged. Investigations can attribute answers to models.

Adjacent Capabilities and Connected Work

This work does not exist in isolation. Routing depends on, and feeds into, the surrounding platform. Ignoring the adjacencies is the most common scoping mistake.

Third-party model risk covers the vendor dependency. Golden datasets supply the evaluation. Distillation is a related cost lever. Prompt versioning manages the variants. Naming these adjacencies upfront keeps the work scoped and helps leadership see behavioural difference as the risk.

The common mistake is treating each adjacency as someone else's problem. The secondary evaluation is your problem. The prompt variants are your problem. The degradation signal is your problem. Pretend otherwise and an outage will be covered while your answers change. Own the adjacencies you depend on, partner with the teams that hold them, and share the deltas.

Conclusion

Fallback patterns come from infrastructure, where the replacement is an identical component doing the same work. Models are not like that. A secondary formats differently, refuses a different set of requests, and interprets the same prompt differently, so during a failover you are serving from a system nobody evaluated for the task, and the responses look exactly like normal ones. Availability is preserved and quality is unknown. Run your evaluation suite against every routing target, maintain prompt variants per model, mark responses served from a fallback, and decide per path whether failing closed is better than serving degraded.

Key Takeaways:

A secondary model is a different system, not an equivalent replacement
Prompts tuned for one model behave differently on another
Fallback degradation is invisible in the response unless it is marked

Designing routing well requires evaluating the alternatives. When done correctly, it produces:

Quality differences known before the outage
Degradation visible to consumers and operators

The AI Product Playbook: Launch Faster, Scale Smarter, Fund with Confidence

Launch faster, scale smarter, and approach funding with greater confidence.