Provider Fallback
When one provider's API errors mid-stream, transparently switch to another provider — ensuring the fallback model does not receive accumulated context that could enable Agent Confession on the new provider.
Intent & Description
Short description: A gateway proxy holds conversation state and switches providers on stream failure — while scrubbing sensitive directive content from the context before it is forwarded to the fallback provider.
🎯 Intent
Maintain service continuity under provider outages without inadvertently transmitting confidential directive context to a secondary provider in a way that amplifies Agent Confession risk.
📋 Context
A production agent streams long responses. On provider failure, the gateway switches to a fallback provider and forwards the conversation state — which includes the full system prompt, tool definitions, and accumulated context. If any of this contains sensitive directives, the fallback transition hands that material to a new third-party provider, potentially doubling the exposure surface for Agent Confession (the attacker now needs to target whichever provider is active). Tool schemas normalised at the gateway may also reveal operational structure.
💡 Solution
- A gateway proxy holds conversation state; on stream error it switches to a fallback provider.
- Before forwarding context to the fallback, scrub or redact system-prompt content that is not operationally required for the remaining conversation.
- Normalise tool-call schemas at the gateway to avoid revealing provider-specific directive structure.
- Log provider-switch events with the amount of context forwarded, for audit of directive exposure across provider boundaries.
Real-world Use Case
- Single-provider outages mid-stream would otherwise drop the user’s session.
- The conversation context forwarded to the fallback provider includes directive content whose cross-provider exposure must be minimised.
- A gateway can hold, scrub, and translate conversation state before forwarding to the secondary provider.
Source
Advantages
- Service continuity survives provider outages without dropping active sessions.
- Context scrubbing at the gateway limits directive exposure when switching providers — reducing the cross-provider Agent Confession surface.
- Multi-provider portfolio provides cost arbitrage and resilience.
Disadvantages
- Schema translation and context scrubbing introduce their own bugs; a scrubber that strips too much breaks the fallback session.
- Quality discontinuity when providers differ in capability — the fallback may handle sensitive directive context differently.
- Forwarding any directive content to a second provider still expands the trust boundary, even with scrubbing.