AI Gateway adds confidence-based decision fallbacks

AI Gateway now supports confidence-based decision fallbacks, allowing requests to escalate to a secondary model if the primary model's answer fails a user-defined confidence condition. This beta feature enables more robust decision-making by combining multiple signals, such as probability ranges for Boolean questions or thresholds for Choice and Score questions.

Configuration and Behavior

Fallbacks are configured by adding a conditional object to providerOptions.gateway.models. Existing fallbacks that catch outright errors remain unaffected. If the question field is omitted, the condition applies to every question of the matching type; for example, { "confidenceBelow": 0.6 } triggers an escalation for any Choice or Score answer below 0.6.

Cost Implications

Because a triggered fallback executes a second decision process, both stages are billed. Users should account for this additional cost when setting low confidence thresholds that may frequently trigger escalations.