Model Fallback in Amazon Bedrock AgentCore Gateway - What Triggers It, What the Circuit Breaker Skips, and Why It Stops at the First Byte

First Published: 2026-11-01
Last Updated: 2026-11-01

Starting October 22, 2026, the inference endpoints of the Amazon Bedrock AgentCore Gateway now support model fallback within gateway rules. A rule selects requests by the model the caller requested, forwards them to a destination, and if the destination fails, attempts to use one or two pre-configured alternative destinations in sequence. A built-in circuit breaker will skip destinations that are repeatedly failing. The caller sends a single request and receives a single response.

This article details which layer—the caller's SDK, the gateway rule, the circuit breaker, or the destination—makes the decision to route a failed request to an alternative destination, and how that decision is reported back to the caller. This article presents this information in layered tables, along with tables of failures that trigger a fallback and those that do not. There are three possible outcomes: a single response (a success, or an error that matches no trigger), the last response (an error), or a partial response followed by an error. This article distinguishes between information explicitly stated in the documentation and information that is not found in the documentation. Conclusions drawn from combining information across multiple documents are presented as this article's reading.

This article is neither an introduction to the AgentCore Gateway's functionality nor a step-by-step configuration guide. It also does not recommend specific models for use as alternatives. Pricing information is not included. The information presented here was verified as of October 30, 2026.

Related articles on this site:

1. Scope and Assumptions
2. How to Read the Tables
3. What Happens During a Single Call
4. Failures Triggering Fallback, and Failures That Do Not
5. What the Circuit Breaker Skips
6. The First-Byte Boundary
7. What the Caller Sees and Does Not See
8. What This Setup Showed
9. Discrepancies and Omissions in the Documents
10. Failure Modes
11. Frequently Asked Questions About Model Fallback in AgentCore Gateway
12. Summary
13. References

1. Scope and Assumptions

This section first defines the intended audience, the scope of this article, the consulted materials, and clarifies distinctions between similarly named components. The key conclusions are as follows:

Fallback is a setting of gateway rules for inference targets. These rules select requests based on the matchModels criteria, determine the destination using the routeToTarget action, and within that action, list one or two alternative destinations in the fallback block.
The default triggers are 5XX status codes, along with three error types: CONNECTION_ERROR, FIRST_BYTE_TIMEOUT, and UNROUTABLE. This enumeration is found in the API Reference documentation and is not present in the body of the DG. When triggerOn is specified, only the listed triggers will be activated. 429 is not included in the default trigger list. Throttling by the gateway itself does not initiate a fallback.
If firstByteTimeoutMilliseconds is not specified, no timeout is configured. Therefore, this article reads that, even if FIRST_BYTE_TIMEOUT is included in the default trigger list, this trigger will not activate unless a timeout value is explicitly defined. Connection-stage timeouts are considered part of the default trigger CONNECTION_ERROR.
triggerOn and timeout configurations are not inherited from the primary destination to alternative destinations. If triggerOn is omitted for an alternative destination, the default triggers will be used for that alternative.
The circuit breaker is enabled by default and maintains state for each combination of target and model. Repeated failures for a particular combination will cause it to be skipped, but the last remaining destination in the chain will not be skipped. The circuit breaker does not count UNROUTABLE. The number of failures or percentage required to open the circuit, the duration for which the circuit remains open, and the percentage of requests allowed through when the circuit is half-open were not found in the documentation.
Fallbacks can only occur before the gateway sends the first byte of the response back to the caller. If a failure occurs after the stream has begun, the caller will receive a partial response and an error, and retry attempts become the caller's responsibility.
The caller's response does not indicate which destination provided the response. The destination that responded can be verified by examining the routing logs delivered by the gateway. Each attempt is counted as one gateway invocation.
Clients invoking InvokeModel receive the gateway's last error in the format of Amazon Bedrock, and the SDK's built-in retry behavior applies to it (the DG says it retries throttling and transient server errors and does not retry validation errors). No statement was found stating that the SDK's retries and the gateway's chain overlap; Section 7.4 discusses how they interact as this article's reading.

1.1 Intended Audience

This article is intended for developers and platform engineers who are operating applications that call multiple models or providers through a single interface, and who are considering implementing fallback rules for the AgentCore Gateway's inference endpoints. Specifically, this article targets readers who believe that at least one of the following statements is true:

By implementing rules, requests will automatically be routed to an alternative model in the event of throttling or errors.
In the event of response timeouts, requests will be routed to an alternative model by default.
Client-side retry logic will no longer be necessary.
It will be clear from the response which model provided the answer.
Even if the destination model fails mid-stream, the gateway will continue processing the request with an alternative model.

However, the validity of each of these assumptions depends on how triggerOn and timeouts are configured, at which stage failures occur, and what information the caller is monitoring.

1.2 Scope and Exclusions

This article addresses the following topics:

Rules, evaluation order, destination, and model selection.
Conditions for fallback activation, and failures that do not trigger fallback.
Circuit breaker states, and what the circuit breaker will and will not skip.
The first-byte boundary, and failure scenarios during stream processing.
What is visible to the caller, what is not, its relationship to SDK retries, and how attempts are counted.
Discrepancies and omissions in the documentation.

This article does not cover the following. This article will not include pricing information, including gateway costs and model pricing.

Client-side retry design (SDK retry modes, backoff and jitter, retry quotas), application circuit breakers, and stream recovery. These topics are covered in the existing LLM Inference Resilience Patterns on AWS.
General operation of AgentCore. These topics are covered in the existing Amazon Bedrock AgentCore Production Operations Guide.
Differentiation between the three target types (MCP, HTTP, Inference) for AgentCore Gateway. These topics are covered in the existing Agent Interoperability Architecture on AWS.
What a translating gateway drops across different API dialects. These topics are covered in the existing What a Translating LLM Gateway Drops.
A comprehensive list of runtime exceptions for Amazon Bedrock. These topics are covered in the existing Amazon Bedrock Errors and Exceptions Reference.
Design of inference profiles and cross-Region inference. These topics are covered in the existing Amazon Bedrock Cross-Region Inference and Data Residency Design.
Model performance and selection of appropriate substitute models.
Where output formatting is guaranteed, a topic covered in the related article, Claude Structured Outputs on Amazon Bedrock.
Where search scope is limited, and the Web Search target of AgentCore Gateway, a topic covered in the related article, Where a Web Search Domain Filter Is Enforced.

1.3 Verification Date and Reviewed Materials

The information was verified on October 30, 2026. The materials reviewed are as follows. In this article, the names of these materials will be abbreviated as follows.

Amazon Bedrock AgentCore Developer Guide (hereinafter referred to as DG): Pages on Model routing and fallback (the DG fallback page), Add rules to a gateway (the DG rules page), Configuration bundle propagation and observability (the DG propagation page), Session stickiness for weighted rules, Rate limit best practices, Use the Amazon Bedrock InvokeModel API with inference targets (the DG InvokeModel page), Inference connector targets (the DG connector page), Inference provider targets (the DG provider page), Core concepts, AgentCore generated gateway observability data (the DG observability page), Quotas, and Release notes for Amazon Bedrock AgentCore.
Amazon Bedrock AgentCore Control API Reference (hereinafter referred to as API Reference): Fallback, FallbackTarget, FallbackTriggers, CircuitBreakerConfiguration, StaticRoute, WeightedRoute, CreateGatewayRule, UpdateGatewayRule, GetGatewayRule, ListGatewayRules, CreateGateway.
AWS CloudFormation Template Reference (hereinafter referred to as CFN pages): The AWS::BedrockAgentCore::GatewayRule resource, and its pages on Fallback, FallbackTarget, FallbackTriggers, and CircuitBreakerConfiguration.
AWS SDKs and Tools Reference Guide: The page on Retry behavior (the SDK retry page).
What's New announcement: AgentCore Gateway now supports Bedrock InvokeModel APIs and model fallback (October 22, 2026).

This article references specific sentences from the documentation. Occasionally, the DG and API Reference materials describe the same topic in different sentences, resulting in variations in the scope covered depending on the page.

1.4 Distinct Entities with Similar Spellings

This article differentiates between similar terms as follows:

Target: In this article, "target" refers to a specific type of target in the AgentCore Gateway – the inference target. The same gateway can also accommodate targets for MCP and HTTP. An Amazon Bedrock inference profile is one type of model that an inference target sends requests to, but it is not a target itself. AgentCore Gateway's Web Search is a separate target, and is not covered in this article.

Fallback: In this article, "fallback" refers to a destination switch determined by the fallback block in the gateway's rules. It is distinct from a fallback implemented directly in the application's code, where the application itself calls a different model. The fallbacks parameter (beta) in the Claude API functions by re-executing the same request with a different Claude model when the initial model declines to respond (refusal). It is not a mechanism for switching destinations due to failures or throttling. The Claude documentation on "Refusals and fallback" describes this server-side fallback as a beta feature for Claude API and Claude Models on Google Cloud (Anthropic managed), and notes that on infrastructure without server-side fallback, the SDK middleware should be used as a substitute.

Circuit Breaker: In this article, "circuit breaker" refers to a server-side circuit breaker enabled by the circuitBreaker setting in the gateway's rules. It is distinct from circuit breakers implemented in the application itself. It is also different from the circuit-breaking functionality used by boto3 for retry quotas, and from the circuit breakers used in Amazon ECS deployments.

Caller and Destination: This article refers to the application or SDK that calls the gateway as the "caller." The destination to which the gateway sends requests is referred to as the "destination," which consists of a target and a model. The DG fallback page also defines the destination as a combination of a target and an optional model.

A destination is a target, optionally with a specific model.

1.5 Inference Endpoint Entry Point

The inference endpoint for each gateway is located at the /inference path. According to the DG InvokeModel page, the gateway determines the dialect of the caller based on the path of the request. There are four supported dialects: InvokeModel (including InvokeModelWithResponseStream), OpenAI's Chat Completions and Responses, and Anthropic's Messages. Amazon Bedrock's Converse and ConverseStream are not mentioned in the DG InvokeModel page, nor are they referenced in the What's New announcements or release notes. While the matchPaths section in the DG rules page includes the reserved prefix /converse, this relates to the HTTP target rules.

The caller authenticates with the gateway, not with Amazon Bedrock. The DG InvokeModel page states:

You authenticate to the gateway, not to Amazon Bedrock. Your credentials need
bedrock-agentcore:InvokeGateway on the gateway.

The gateway performs translations between dialects. The DG InvokeModel page lists the following translations: relocating the model ID, rewriting request paths, adjusting the anthropic-version header for Anthropic routes, and re-framing streams between the Amazon Bedrock event-stream format and server-sent events (SSE). However, the payload schema is not transformed.

The gateway does not convert one payload schema into another.

These two aspects influence how fallback chains are configured. They are discussed in Section 4.6 and Section 10.

2. How to Read the Tables

This section defines the columns in the tables from Sections 3 through 9, as well as how to describe the state of the sources. The terminology used for the columns and state descriptions is shared with the other two articles in the same series.

2.1 Columns

The tables in this article begin with a Layer column, followed by subsequent columns. In some tables, instead of Layer, the first column may indicate the type of failure or the trigger.

What It Decides: What that layer determines.
What the Caller Receives: What the caller receives as a result of that layer's decision.
Where It Is Written: The source of the sentence in this row. Specify the document and page.
No Statement Found: Instances where, after searching the materials described in Section 1.3, using different search terms and document categories, no relevant sentence was found. Use — when there is no such item.

A non-empty No Statement Found entry does not indicate that the matter is nonexistent or will not occur.

2.2 Four States

The state of the sources is described using the following four terms:

No statement was found: In the text, this is written as no statement was found, accompanied by the scope of the search.
The documentation does not list: This indicates that the item is not included in a list or table. In the text, this is written as the documentation does not list.
The sources disagree: In the text, this is written as the sources disagree, and both statements, along with their sources, are presented. This article does not definitively identify either source as incorrect.
Inference: In the text, this refers to this article's reading, and is not a statement found in any of the source materials. This is indicated in the same sentence.

2.3 Layer Terms

This article refers to layers using the following five terms:

Caller: The application that calls the gateway and its associated SDK.
Gateway: The component that evaluates rules, forwards requests to the destination, and returns responses to the caller.
Rule: A configuration that determines which requests are sent to which destination and which failures trigger the next destination.
Circuit breaker: A component that maintains status for each destination and determines whether to skip a destination before attempting it.
Destination and Provider: A destination consists of a target and a model. The provider is the entity or service that operates the model at that destination.

This article maintains a clear distinction between the caller and the destination. The caller is located before the gateway, while the destination is located after the gateway.

2.4 Defaults, Examples, and Overrides

This section describes trigger and timeout values, categorized into three types:

Defaults: Values used when settings are not explicitly specified. In this article, these values are described in the API Reference documentation.
Examples: Values defined in the DG example settings. These are not default values.
Overrides: Values specified using the triggerOn parameter. These values replace the default values, rather than being added to them.

2.5 Document Text and Announcement Text

This article will use sentences structured around the page name as the subject when describing the content of DG pages, the API Reference, and CFN pages. Announcements in the What's New section will include both the announcement name and the date in the same sentence; they are not cited as specification. The terminology used in announcements and DG documentation will be listed in Section 9.

3. What Happens During a Single Call

This section describes what is determined in the gateway in sequence when the caller makes a single call. To state the conclusions first, there are four:

The gateway evaluates rules in order of priority, using the first matching rule for each type of action.
The routeToTarget action determines the primary destination. If the destination model is pinned, the gateway can send a model different from the one requested by the caller.
The gateway traverses the chain sequentially, checking the circuit breaker for each destination, and attempts each destination only once. Responses that match no trigger are returned directly to the caller.
If all destinations fail, the caller receives the last response. The DG describes this as the last error, while the API Reference documents it as the last response.

3.1 Rule Evaluation

The DG rules page specifies the order of rule evaluation as follows:

The gateway evaluates rules in ascending priority order (lower numbers first).
The gateway resolves each action type independently using first-match semantics.

Each rule is assigned a priority, represented by an integer from 1 to 1,000,000, and must be unique in the gateway. Rules without any conditions match all requests. The maximum number of rules per gateway is 20.

The matchModels condition selects requests based on the model specified by the caller. The value can be a model ID or a glob pattern; if the operator is omitted, it defaults to StringLike. The DG rules page states:

Because matchModels values are glob patterns by intent, an omitted operator keeps
* wildcards active; use StringEquals for an exact, literal match.

The default operator for matchPrincipals is StringEquals, which is the opposite of the default for matchModels. In rules using matchModels where the operator is omitted, any * characters in the value function as wildcards.

3.2 Destination and Model

The routeToTarget action has two modes of operation: staticRoute and weightedRoute. staticRoute directs traffic to a single target, while weightedRoute distributes traffic to two targets based on weighting. For inference targets, both staticRoute and each weightedRoute leg allow you to specify the model to be sent. If no model is specified, the gateway will send the model requested by the caller.

When no rules are defined, the gateway selects the target based on the model ID. If multiple targets match the same model, the DG fallback page states that a rule can pin the target.

When several targets match the same model, default routing can send the request
to any of them; a rule pins it to the target you choose.

The method for distributing traffic between targets that match with equal strength when no rules are defined is described differently on the DG connector page and the DG provider page. This is discussed in Section 9.1.

3.3 Walking the Chain

The fallback block should be placed in the routeToTarget action. This block contains the following four items:

targetFallback: One to two alternative destinations to try after the primary destination.
triggerOn: The condition for transitioning from the primary destination to the next alternative destination.
firstByteTimeoutMilliseconds: Timeout for the first byte received from the primary destination.
circuitBreaker: Circuit breaker settings.

Each alternative destination can also have its own triggerOn condition and timeout settings.

The DG fallback page describes the fallback flow as follows:

The gateway walks the chain in order. For each destination it checks the circuit
breaker, attempts the destination once, and either returns the response to the
caller or, if a trigger fires, advances to the next destination. If every
destination fails, the caller receives the last error.

The gateway attempts each destination only once. This sentence does not specify attempting the same destination repeatedly. Responses that match no trigger, even if they are error responses, are returned directly to the caller. The DG fallback page describes triggerOn as follows:

A response that matches none of them returns to the caller unchanged.

The API Reference's Fallback page describes the same flow as follows:

The gateway tries each target in order until a response doesn't match that
target's triggers, and returns that response to the caller. If every target
fails, the gateway returns the last response.

Destinations must not be duplicated within a rule. The primary destination and each alternative destination must each use a different combination of target and model. Different models for the same target can be listed as separate destinations.

Every destination in a rule must be distinct. The primary and each fallback
destination must name a different combination of target and model; the gateway
rejects a rule that repeats one.

3.4 Weighted Routes

When the primary route is designated as weightedRoute, fallback configurations are applied to each leg. The weightedRoute itself does not have a route-level fallback. The DG fallback page states:

A leg with no fallback returns the error to the caller when it fails. The circuit
breaker is per destination and independent of the weights.

The DG Session stickiness page describes configurations that use weighted rules to ensure that a session is consistently directed to the same target. However, no statement was found describing how a fallback is handled when the target associated with a session fails. This is the no statement was found state; the search covered the DG Session stickiness page, the DG fallback page, and the DG rules page.

3.5 Layer-by-Layer Table

This table outlines what each layer determines and what the caller receives during a single invocation.

Layer	What It Decides	What the Caller Receives	Where It Is Written	No Statement Found
Caller's SDK	Whether to send the request to the gateway and whether to retry the last error.	A typed exception or a successful response.	DG InvokeModel page, Errors section.	A statement that SDK retries and gateway chaining overlap.
Gateway Rules	Evaluates rules in order of priority and determines which rule's `routeToTarget` to use.	The response from the destination selected by the rule.	DG rules page, Rule resolution section.	—
Rule `fallback`	Determines which responses and errors trigger a transition to the next destination, and in what order those destinations are attempted.	A single response, or the last response.	API Reference, `Fallback` and `FallbackTarget`; DG fallback page.	—
Circuit Breaker	Determines whether to skip a destination before attempting it.	The response from the destination after the skipped one. There is no indication that a skip occurred.	API Reference, `Fallback`; DG fallback page, Circuit breaker section.	Numerical values for when the circuit opens, how long it remains open, the percentage of requests allowed through in a half-open state, and where the state is maintained.
Destination and Provider	Determines whether to respond to the request or return an error.	The response from the destination. The gateway returns the response in the caller's dialect.	DG InvokeModel page, Errors section.	—
Gateway Observability	Records which destination responded and whether the circuit breaker skipped a destination.	Not returned to the caller.	DG fallback page, Observability and billing section.	Field names in routing logs.

4. Failures Triggering Fallback, and Failures That Do Not

This section outlines which failure scenarios trigger a fallback, and which do not, categorized by default settings, examples, and overrides. To begin, there are six key conclusions:

Default triggers are 5XX errors and three other error types. This listing is present in the API Reference and the CFN pages, but the body of the DG fallback page does not list the default triggers.
When triggerOn is specified, only the specified triggers will activate. It does not supplement the default settings.
429 is not a default trigger. Throttling by the gateway itself, regardless of what is specified in triggerOn, will not trigger a fallback.
If firstByteTimeoutMilliseconds is omitted, no first-byte timeout is enforced, and this article reads that FIRST_BYTE_TIMEOUT will not trigger a fallback. Connection-stage timeouts will fall under CONNECTION_ERROR.
triggerOn and timeout settings are not inherited by alternative destinations. If triggerOn is omitted on an alternative destination, the default triggers will be used; if the timeout is omitted, there is no timeout.
There are failure scenarios that the gateway creates on the fly for which fallback behavior is documented, and others for which it is not.

4.1 Default Triggers

The API Reference page for Fallback describes the default trigger as follows, and the Fallback page for CFN states the same.

If you omit triggerOn, the gateway falls back on 5XX status codes and on the
CONNECTION_ERROR, FIRST_BYTE_TIMEOUT, and UNROUTABLE errors. If you specify
triggerOn, only the status codes and errors that you list trigger a fallback.

The body of the DG fallback page does not list the default triggers. It only states, at the end of the example configuration description, that the alternative destinations use the default.

The two fallback destinations take the service defaults.

The following three error conditions are defined on the DG fallback page.

CONNECTION_ERROR — The gateway could not establish a connection to the destination
(connection refused, reset, or timed out). The attempt never reached the provider.
FIRST_BYTE_TIMEOUT — The destination accepted the connection but did not return the
first response byte within firstByteTimeoutMilliseconds.
UNROUTABLE — The gateway could not route to a serving target (the target is missing,
or it does not support the request’s API path). This is a deterministic,
gateway-side condition: it advances the chain but does not feed the circuit breaker.

Status codes can be specified with one to ten values. These values can be precise codes, such as 429, or classes, such as 5XX, and are limited to the 4xx and 5xx ranges. The DG fallback page also notes that successful responses are not eligible for fallback.

Fallback applies only to error responses; 2xx and 3xx responses are never
fallback-eligible.

4.2 `triggerOn` Supersedes the Default

When you specify triggerOn, only the defined status codes and errors will trigger a fallback. For example, if you define a rule with only statusCodes set to 429, this article reads that a 5XX response or a connection error will not trigger a fallback. This is consistent with the wording in the API Reference in Section 4.1, which uses the term "only."

The example on the DG fallback page lists the following for the primary triggerOn: status codes 5XX and 429, all three error types, and a 5-second first-byte timeout. The DG explains this example configuration as follows:

The primary falls back on 429 and 5XX, on a connection error or first-byte timeout,
and on an unroutable target, with a 5-second first-byte timeout.

This sentence describes the configuration used in the example. The default configuration, as described in the API Reference in Section 4.1, does not include 429 and does not specify a timeout duration. The DG example also notes that the target and model names are used as examples.

The target and model names in this example are illustrative.

This article does not present the model names in the example as recommendations. In this article's descriptions, the model will be represented as <model-id>.

4.3 429 and Throttling

A 429 response from the destination will only trigger a fallback when triggerOn includes 429 or 4XX in its statusCodes. It is not a default trigger.

Throttling performed by the gateway itself will not trigger a fallback. The API Reference page for FallbackTriggers states this, and the CFN page for FallbackTriggers contains the same wording.

Only 4XX and 5XX codes are supported. Throttling by the gateway itself doesn't
trigger a fallback.

The DG InvokeModel page states that rate limiting rejections for gateway tokens are returned as throttling errors in the caller's format.

Governance outcomes surface the same way: a Amazon Bedrock AgentCore Policy denial
is an access-denied error, and a token-rate-limit rejection is a throttling error,
each in the client’s format.

The DG Rate limit best practices page states that rate limiting rejections for tokens occur before the request reaches the model.

If the estimate exceeds the remaining budget, the request is rejected with an HTTP 429
response before it reaches the model.

Based on these statements, this article reads that requests rejected due to gateway token rate limiting are not routed to another destination; instead, the caller receives a throttling error. This is this article's reading; no source has a statement that connects rate limiting with fallback mechanisms. The specific paths affected by token rate limiting are described differently across various documents, a topic addressed in Section 9.5.

4.4 Two Timeouts

There are two timeouts related to fallback: the connection-stage timeout and the first-byte timeout.

The connection-stage timeout results in a CONNECTION_ERROR. The DG definition in Section 4.1 lists connection refusal, reset, and timeout as possibilities. Therefore, even in a rule without triggerOn, a request that times out while attempting to connect to a destination will proceed to the next destination.

The first-byte timeout is determined by the firstByteTimeoutMilliseconds setting. The API Reference page for Fallback states:

The maximum time, in milliseconds, from sending the request to the route's target
until the first byte of the response body arrives. If the time expires, the gateway
abandons the request and treats it as a FIRST_BYTE_TIMEOUT error. If you don't
specify a value, no timeout is enforced.

The value range is between 50 and 900,000 milliseconds. The DG fallback page describes the purpose of this setting as ensuring that fallback is triggered even if the provider stalls instead of returning a clean error.

Here, statements from two sources are combined. The default trigger list includes FIRST_BYTE_TIMEOUT. However, if firstByteTimeoutMilliseconds is omitted, no timeout is configured. Based on these two points, this article reads that if firstByteTimeoutMilliseconds is not specified in a rule, FIRST_BYTE_TIMEOUT will not be triggered, and fallback will not occur, even if the destination accepts the connection but then fails to send a response. This is this article's reading; this combined statement is not found in the documentation. This article reads that in such a scenario, the gateway call timeout quota and the timeout settings in the caller's SDK determine the request's termination point. However, no statement was found describing the relationship between the fallback for the inference endpoint and this quota. This is the no statement was found state; the search covered the DG Quotas page, the DG fallback page, and the API Reference for Fallback. The DG Quotas page states that the gateway call timeout is 15 minutes. While 900,000 milliseconds is equivalent to 15 minutes, no statement was found explicitly linking these two values.

4.5 Settings That Are Not Inherited

The triggerOn setting and timeout settings are not inherited from the primary configuration to alternative destinations. The API Reference page for FallbackTarget states that triggerOn is configured as follows:

The status codes and errors from this target that cause the gateway to send the
request to the next fallback target. This setting isn't inherited from the route.
If you omit it, the gateway uses the default triggers: 5XX status codes and the
CONNECTION_ERROR, FIRST_BYTE_TIMEOUT, and UNROUTABLE errors.

The same applies to firstByteTimeoutMilliseconds. The FallbackTarget page also includes the following statement regarding this setting, and states that if it is omitted, no timeout is enforced.

This setting isn't inherited from the route.

These two statements apply to the rule where the primary configuration includes 429 and timeout settings, while the alternative destinations' configuration remains empty. In this rule, this article reads that if alternative destination 1 returns a 429 error, it will not proceed to alternative destination 2, and the 429 error will be returned to the caller. This article also reads that even if alternative destination 1 accepts the connection but then stops, it will not proceed to alternative destination 2, as there is no timeout configured for alternative destination 1. The DG example configuration also does not include triggerOn or timeout settings for the alternative destination. The release notes state that timeout settings are configured on a per-destination basis.

You configure which failure types trigger fallback and set a per-destination
first-byte timeout.

4.6 Failures Created by the Gateway

In addition to failures returned by the destination, there are failures that the gateway creates on the fly. For some, the documentation describes the relationship with fallback, while for others it does not.

Target not found, or the target does not support the API path of the request: Returns UNROUTABLE, which is a default trigger. The circuit breaker does not count it.
Gateway token rate limiting: Results in a throttling error and does not utilize fallback (this article's reading in Section 4.3).
AgentCore Policy rejection: Returns an access denied error. No statement was found describing its relationship with fallback mechanisms.
Incompatible dialect combinations, or the payload is rejected by the destination: The DG InvokeModel page states that the gateway returns a 400 error in the caller's dialect. For the InvokeModel client, this is returned as a Bedrock ValidationException.

When the gateway cannot serve a request — an unsupported dialect pair, or a payload
that the destination rejects — it returns a 400 error in the client’s dialect (for
an InvokeModel client, a Bedrock ValidationException).

An InvokeModel request with guardrails specified is routed to the translation path: The DG InvokeModel page states that the gateway rejects the request with a 400 error.

400 ValidationException: Cannot translate a Bedrock InvokeModel request with a
customer guardrail configured.

The last two points impact how fallback chains are configured. The gateway does not transform the payload schema. Therefore, this article reads that if you place a model as a fallback that uses a different payload schema than the primary model, attempts to reach that fallback will result in a 400 error. The DG example chains use models with the same lineage name for all three destinations. This article also reads that for InvokeModel requests with guardrails, attempts to reach the fallback on the translation path will also result in a 400 error. No statement was found that specifies whether these two 400 errors are treated as triggerOn status codes, as UNROUTABLE, or in some other way. This is the no statement was found state; the search covered the DG fallback page, the DG InvokeModel page, and the API Reference for FallbackTriggers.

4.7 Trigger Table

This table details, for each failure type, whether it fires with the default triggers, what happens when triggerOn is written, and what the caller receives.

Failure	With Default Triggers	With triggerOn	What the Caller Receives	Where It Is Written	No Statement Found
Destination 5xx	Fires	Fires only when written	The response from the next destination. If it's the last destination, the last response.	API Reference, `Fallback`	—
Destination 429	Does not fire	Fires only when `429` or `4XX` is written	If it doesn't fire, the original 429 is returned.	API Reference, `Fallback`; DG example	Does 429 count towards the circuit breaker?
Destination other 4xx	Does not fire	Fires only when written	If it doesn't fire, the original 4xx is returned.	API Reference, `FallbackTriggers`	—
Connection Refusal, Reset, Timeout	Fires. Enters `CONNECTION_ERROR`.	Fires only when written	The response from the next destination.	DG fallback page	—
No first byte received after connection	Does not fire unless timeout is written (this article's reading)	Fires when a timeout is written and `triggerOn` includes `FIRST_BYTE_TIMEOUT`.	If it doesn't fire, it continues to wait for the destination's response.	API Reference, `Fallback`	Time when the request ends if it doesn't fire.
Target not found, path not supported	Fires. Enters `UNROUTABLE`.	Fires only when written	The response from the next destination.	DG fallback page	—
Gateway's own throttling	Does not fire	Does not fire	Throttling error	API Reference, `FallbackTriggers`	—
Gateway-generated 400 (dialect, payload, guardrail)	—	—	A 400 error in the caller's dialect (a Bedrock `ValidationException` for an InvokeModel client)	DG InvokeModel page	How it's handled when created in a chain.
2xx and 3xx responses	Does not fire	Does not fire	The response itself	DG fallback page	—
Failure after sending the first byte to the caller	Does not fire	Does not fire	Partial response and error	API Reference, `Fallback`; DG fallback page	Error format

What Triggers a Fallback and What Does Not

5. What the Circuit Breaker Skips

This section describes the unit of state maintained by the circuit breaker, what it skips, and what it does not skip. To summarize, there are five key points:

The circuit breaker is enabled by default and can be disabled by setting the circuitBreaker mode to DISABLED. Only an enabled or disabled state can be configured.
There are three states, kept for each combination of target and model.
The circuit breaker does not count UNROUTABLE.
The circuit breaker will not skip the last remaining destination in a chain.
The numerical values for the conditions that trigger the circuit breaker to open, the duration it remains open, the percentage of traffic allowed when half-open, and the location where the state is stored could not be found in the documentation.

5.1 Defaults and Configuration

The CircuitBreakerConfiguration section of the API Reference specifies two possible values for the mode setting: ENABLED and DISABLED, with the default being ENABLED. The CircuitBreakerConfiguration only includes the mode setting. There are no settings related to the conditions under which the circuit breaker opens, or the duration for which it remains open, either in the API Reference or in the CFN pages.

Circuit breaker settings related to fallback functionality are found in the fallback block. There are no circuit breaker-specific settings in the FallbackTarget.

5.2 Three States

The DG fallback page describes how the circuit breaker automatically transitions between three states.

Closed (Healthy): The gateway sends requests to the primary. If the primary's response matches a trigger, the gateway proceeds to the next destination.
Open (Failing): The gateway skips the primary and sends requests directly to the next destination in the chain.
Half-open (Recovering): The gateway forwards only a portion of the requests to the primary.

The explanation for Half-open is as follows:

The gateway lets a small portion of requests through to the primary to check whether
it has recovered; the rest skip the primary and go to the next destination. If the
requests allowed through succeed, the breaker closes and all traffic returns to the
primary. If they fail, the breaker stays open.

5.3 Target and Model Combinations

The Fallback section in the API Reference describes the circuit breaker as follows. The Fallback section on the CFN page uses the same wording.

The circuit breaker tracks each combination of target and model separately. With the
circuit breaker enabled, the gateway skips one or more targets and models that fail
repeatedly, and your requests go to the next target in the chain. The gateway always
tries the last remaining target in the chain. The gateway enables the circuit breaker
by default.

The state is determined not by target alone, but by the combination of target and model. This article reads that if one model associated with the same target repeatedly fails, causing the circuit to open, the destination associated with a different model for that same target will not be skipped.

5.4 Items Not Counted by the Circuit Breaker

UNROUTABLE advances the chain, but the circuit breaker does not count it. The FallbackTriggers page in the API Reference includes the following sentence at the end of the UNROUTABLE description:

These errors don't count toward the circuit breaker.

The DG fallback page describes UNROUTABLE as a condition determined on the gateway side.

No other statement was found that lists items to be counted or not counted as circuit breaker events. Specifically, no statement was found on whether 4xx errors, such as 429 listed in triggerOn, are counted by the circuit breaker. This is the no statement was found state; the search covered the DG fallback page, the API Reference sections on Fallback, FallbackTriggers, CircuitBreakerConfiguration, the CFN pages, and the release notes. In the setup in Section 8.6, a destination that kept returning 4xx errors was attempted 10 times in a row and was not skipped, but this observation alone does not show why.

5.5 The Last Remaining Destination

Of the sentences in the API Reference section described in Section 5.3, the following sentence is part of the circuit breaker explanation:

The gateway always tries the last remaining target in the chain.

The preceding sentence describes skipping destinations that repeatedly fail and proceeding to the next destination. This article reads that sentence to mean that the circuit breaker does not skip the last remaining destination in the chain. It does not mean that alternative destination 2 is always attempted. If the primary destination or alternative destination 1 returns a response that matches no trigger, that response is sent back to the caller, and alternative destination 2 is not attempted.

5.6 When the Circuit Breaker Is Disabled

Even when the circuit breaker is disabled, the chain proceeds in the order in which it was declared. The DG fallback page states:

With the breaker off, the chain still walks in the order you declared.

The DG cites an example of why the circuit breaker might be disabled: continuing to use a less expensive primary while delaying a switch to a more costly alternative destination. This article reads that disabling the circuit breaker means that even a failing primary will continue to be attempted, and those attempts will be counted (Section 7.5).

5.7 What Is Not Written

The following information was not found in the documentation. This is the no statement was found state, and the areas searched included the DG fallback page, the DG rules page, the DG observability page, the API Reference for Fallback, FallbackTarget, FallbackTriggers, CircuitBreakerConfiguration, the CFN pages, the release notes, and the What's New announcements.

Numerical values for the conditions that trigger the circuit to open. This includes the number of failures, the failure rate, and the time window for counting failures. The documentation only mentions the phrase "fail repeatedly."
The duration for which the circuit remains open, and the time until it transitions to the half-open state.
The percentage of requests routed to the primary during the half-open state. The documentation only mentions the phrase "a small portion."
Where the circuit's state is stored: whether it is managed at the gateway level, the Region level, or based on the rules themselves. The release notes only mention the phrase "service-owned circuit breaker."

This article will not contain any speculative estimates regarding these values.

Circuit Breaker States per Target and Model

6. The First-Byte Boundary

This section defines until when fallback may occur, using two meanings of "first byte." To state the conclusions first, there are three:

The "first byte" for a timeout is the time elapsed until the first byte of the response body arrives from the destination.
The "first byte" for the fallback boundary is the point at which the gateway sends the first byte of the response to the caller.
In cases of failure after the stream has begun, the caller will receive a partial response and an error, and any retries will be the caller's responsibility.

6.1 Two "First Bytes"

firstByteTimeoutMilliseconds measures the time from when a request is sent to a destination until the first byte of the destination's response is received (see Section 4.4). This is the time between the gateway and the destination.

The fallback boundary occurs between the gateway and the caller. The Fallback page in the API Reference states:

A fallback can occur only before the gateway sends the first byte of the response to
the caller. After a streaming response starts, the gateway doesn't fall back.

The documentation does not directly correlate these two "first bytes." This article reads the fallback opportunity as ending when the gateway begins sending the destination's first byte of response to the caller. This is this article's reading.

6.2 Handling Failures During Streaming

The DG fallback page describes how to handle failures during streaming as follows:

Once a streaming response has started, the gateway can no longer fall back; the
caller receives the partial response and an error, and must retry.

The partial response received by the caller contains the portion of the response that the failed destination had already sent. The gateway does not attempt to continue the response with a different destination. No statement was found on how the error following the partial response is delivered – for example, whether it arrives as an exception event in the stream of InvokeModelWithResponseStream. This is the no statement was found state; the search covered the DG fallback page, the DG InvokeModel page, and the API Reference for Fallback.

While the existing article LLM Inference Resilience Patterns on AWS addresses stream recovery, this article reads that even with a gateway in place, handling failures during streaming ultimately remains the responsibility of the caller.

6.3 Time After the First Byte

The DG connector page states that there are no service limits regarding stream length and response size.

AgentCore Gateway does not enforce a service-level maximum on response stream
duration or response size.

The DG connector page includes this statement to explain the rationale for implementing a token limit policy. This article reads that, after the first byte is received, neither the fallback timeout nor the fallback mechanism itself takes effect. Therefore, what limits the stream length is the token limit policy and the caller's timeout.

7. What the Caller Sees and Does Not See

This section describes how the fallback result appears from the caller's perspective. To state the conclusions first, there are five:

The caller submits one request and receives one response. There is no indication of which destination provided the response.
The destination that responded, and the fact that the circuit breaker skipped a destination, are recorded in the gateway's routing logs.
The InvokeModel client receives the last error as an Amazon Bedrock error, and the SDK's built-in retry behavior applies to it (throttling and transient server errors are retried; validation errors are not).
This article reads that the SDK retries and the gateway chain overlap by multiplication.
Each attempt is counted as one gateway invocation.

7.1 A Single Response, Without an Indicator

The DG fallback page describes fallback, as viewed from the caller, as follows:

From the calling client’s perspective, fallback is transparent: the client sends one
request and receives one response, from whichever destination served it. The gateway
adds no caller-facing indicator of which destination was used.

The sources do not say that a response header or body field indicates which destination responded. They state that the gateway does not add an indicator to the response. In some dialects, the response body may originally include a field indicating the model's name, such as the model field in Anthropic's Messages responses. This value is part of the provider's response and is not a marker added by the gateway. This article does not recommend using this value to verify the destination. No statement was found in the sources in Section 1.3 that describes what value this field might take after a fallback.

7.2 Routing Logs and Span Attributes

Verify the destination in the gateway's records. The DG fallback page states:

The gateway records fallback and routing decisions in the routing logs it vends. These
logs attribute the destination that actually served each request and note when the
circuit breaker skipped a destination, so you can confirm routing and see why a
destination is being skipped.

No statement was found regarding the field names, formats, and log types for the routing logs. This is the no statement was found state, and the areas searched included the DG fallback page, the DG observability page, and the DG propagation page. While the DG observability page specifies the default log group for gateway logs as /aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/{gateway_id}, it does not mention the terms "inference," "fallback," "routing," or "model." No statement was found on CloudWatch metrics specifically for fallback events. In the setup in Section 8.3, the gateway logs had records with a field named body.routing, and for requests answered by an alternative destination, the reason value was either servedViaFallbackHop:1 or servedViaFallbackHop:2. This is an observation, not a statement in the sources.

The DG propagation page indicates that the resolution results of the rules are output as attributes of the gateway server's span. The attribute for target routing is aws.agentcore.gateway.resolved_target_name, and its description consists of a single sentence:

The name of the resolved target.

The documentation does not specify whether this attribute is the target name actually used after the fallback, or the target name initially selected by the rule. Furthermore, there is no information regarding attributes related to the model name on this page.

7.3 Error Formats and SDK Retries

The gateway returns errors in the caller's dialect. The DG InvokeModel page states the following regarding InvokeModel clients:

An InvokeModel client receives the Amazon Bedrock JSON error body with the
x-amzn-errortype header, so the Amazon Bedrock SDK builds its typed exceptions and
applies its built-in retry behavior, which retries throttling and transient server
errors and does not retry validation errors.

Specifically, if all destinations fail and the last error is returned, the caller's SDK will resend the request to the gateway, provided the error indicates throttling or a temporary server error. Requests are not resent if a ValidationException is returned.

The number of retry attempts performed by the SDK is documented on the SDK retry page. It is important to note that this page's behavior is a new behavior that can be enabled through environment variables.

The behavior described on this page requires opting in until it becomes the default
behavior.

This page specifies a default maximum number of attempts of 3, and explains this as follows:

A max attempts value of 3 means the SDK makes one initial request and up to two retries.

The SDK's retry mode, backoff strategy, and retry quotas are covered in the existing article LLM Inference Resilience Patterns on AWS.

7.4 How SDK Retries and the Gateway Chain Overlap

No direct statement was found describing the overlap between SDK retries and the cascading fallback chains in the gateway. This is the no statement was found state; the search covered the DG fallback page, the DG InvokeModel page, the SDK retry page, and the What's New announcements. The following is this article's reading.

When the SDK retries a last error, that retry results in a new request to the gateway. The gateway then applies its own rules to that request and begins the fallback chain again. Unless the circuit breaker skips a destination, a single request can result in up to three attempts to reach the destination (the primary destination and two alternatives). If the maximum number of SDK attempts is defined as max_attempts, a single application call can result in up to 3 × max_attempts attempts to reach the destination. For example, if max_attempts is 3, the maximum number of attempts would be 9.

This multiplication effect only occurs when the SDK retries the last error. As described on the DG InvokeModel page, throttling and temporary server errors are retried, while ValidationException errors are not. If the circuit breaker skips a destination, subsequent requests may result in fewer attempts.

This article reads that the decision to retain or remove SDK retries is determined by both this multiplication effect and how failures during a stream are handled. The DG fallback page states that fallback mechanisms can eliminate the need for each client to implement its own fallback logic.

Fallback gives you higher availability and more control over cost and model coverage,
without each calling client implementing its own fallback logic.

This passage describes logic related to switching to an alternative model. The responsibility for handling both last-error retries and failures during a stream remains with the caller, as outlined in Sections 6.2 and 7.3.

7.5 Counting Attempts

The DG fallback page describes how to count attempts as follows:

Each outbound attempt the gateway makes counts as a gateway invocation and is billed at
gateway rates. A request that falls back across multiple destinations therefore counts
as one gateway invocation for each attempt, and the circuit breaker reduces this cost by
skipping destinations that are already failing.

According to the circuit breaker table on the DG fallback page, when in the "Closed" state, failed primary attempts before the fallback are also counted. When in the "Open" state, only attempted destinations are counted; the skipped primary is not counted. When in the "Half-open" state, attempts directed to the primary are counted, while other attempts are counted the same way as in the "Open" state. DG states that the model inference cost of each attempt is billed separately by Amazon Bedrock or the model provider, rather than by the gateway. This article does not include amounts.

7.6 Table of Caller-Side Layers

This table outlines what is visible and what is not visible from the caller's perspective.

Layer	What It Decides	What the Caller Receives	Where It Is Written	No Statement Found
Gateway Response	Which destination's response to return	A single response. There is no indication of the destination that provided the response.	DG fallback page	—
Gateway Error	Which last response to return, using the caller's dialect	The Bedrock JSON error and the `x-amzn-errortype` header are sent to the InvokeModel client.	DG InvokeModel page	Shape of errors occurring mid-stream
Caller's SDK	Whether to retry the last error	Either the response after retrying, or a typed exception.	DG InvokeModel page, SDK retry page	A statement that retries and chains overlap.
Routing Logs	Records the destination that responded and any destinations the circuit breaker skipped.	Not returned to the caller.	DG fallback page	Field names, log types
Span Attributes	Provides the name of the resolved target.	Not returned to the caller.	DG propagation page	Whether it is the target after fallback
Attempt Counting	Counts each attempt as a single invocation.	Not returned to the caller.	DG fallback page	—

8. What This Setup Showed

This section details observations made using the demo configuration created on October 30, 2026, separating the values in the sources from the observed values. The values presented here are records of what was observed using this configuration, and are not intended for performance comparisons. Account IDs, gateway IDs, ARNs, endpoint names, and request IDs have been anonymized.

8.1 Setup

The Region is us-east-1. AWS CloudFormation created the execution role for the gateway, a gateway with IAM authorization, the inference target for the bedrock-runtime connector, and a log group for the gateway's APPLICATION_LOGS delivery. A Python script rewrote the gateway rule for each scenario and then deleted it.

The model is Amazon Nova Micro. The caller requested a model ID that matches the matchModels parameter in the rule. To cause failures, the primary pinned a model ID that does not exist, and the alternative destination used the ID of the same model's inference profile. Because this is a different model ID on the same target, it is listed as a separate destination. For the purpose of observing the gateway's behavior, the caller's SDK retry attempts were fixed to a single attempt.

The calls were made 33 times between 09:43 and 09:48 UTC on October 30, 2026. The rules were rewritten for each scenario, and after each rewrite, the script waited 30 seconds before making a call. The gateway's APPLICATION_LOGS were read from CloudWatch Logs 3 minutes, 6 minutes, and about 80 minutes after the calls.

8.2 An Indicator of the Destination That Answered

In both calls without a rule and calls answered by an alternative destination, the combination of header names in the response was consistent. The only values that differed were the date, x-amzn-bedrock-invocation-latency, and x-amzn-errortype in cases of errors. In the response body, in every call that returned a JSON body, the fields were limited to output, stopReason, and usage from Amazon Nova. In this setup, the response did not show which destination or which model answered. This aligns with the DG fallback page sentence in Section 7.1.

8.3 Contents of Routing Logs

The gateway logs contained one record per request, each with a body.routing field. There were 33 records corresponding to the 33 calls. A record for a request answered by an alternative destination had the following form. The rule ID is masked.

{
  "outcome": "QUALIFIED",
  "matchType": "QUALIFIED_OVERRIDE",
  "reason": "servedViaFallbackHop:1",
  "requestPath": "/model/amazon.nova-micro-v1:0/invoke",
  "requestedModelId": "amazon.nova-micro-v1:0",
  "resolvedModelId": "us.amazon.nova-micro-v1:0",
  "routingRuleId": "<rule-id>",
  "targetName": "bedrock"
}

In this setup, the routing field contained the following items: outcome, matchType, reason, requestPath, requestedModelId, resolvedModelId, routingRuleId, targetName, and score. The reason field was present only in records for requests answered by an alternative destination, and its value was either servedViaFallbackHop:1 or servedViaFallbackHop:2. The resolvedModelId was the model ID of the destination that answered. Records for requests with no rule had an outcome of ROUTED, a matchType of SINGLE_MATCH, and included a score.

The body.log section of the logs contained four different types of messages. The IDs are masked.

Executing Http request for target <target-id> - RequestId: <request-id>
Model routing succeeded
Action resolved from rule <rule-id> (priority 100.0): Route to target: targetName=bedrock, ruleId=<rule-id>
An error occurred while executing Http request for target <target-id> - RequestId: <request-id>

The first type of message appeared 61 times and matched the total number of attempts to destinations across the scenarios. This article reads this message as a record of each attempt to a destination. The fourth type of message appeared only in the scenarios that set a first-byte timeout. These field names and messages were observed in this setup. Because the DG does not list the routing log fields, the no statement was found in Section 7.2 remains unchanged. It is possible that these items may change in the future.

8.4 Trigger Conditions and Inheritance

Here are the results for each scenario, as follows. The number of attempts to reach the destination is the count of the first log message in Section 8.3.

Scenario	Rule	Attempts per Call	What the Caller Received	`reason`
S0	No rule	1	200	—
S1	A model ID that does not exist pinned on the primary; `triggerOn` omitted	1	400 `ValidationException`	—
S2	S1 plus `triggerOn` with `4XX`, `5XX`, and the three errors	2	200	`servedViaFallbackHop:1`
S5	Primary moves on at `4XX`; alternative destination 1 also pins a model ID that does not exist and omits `triggerOn`; alternative destination 2 has a valid model	2	400 `ValidationException` from alternative destination 1	`servedViaFallbackHop:1`
S5b	S5 with `4XX` also written on alternative destination 1	3	200	`servedViaFallbackHop:2`

In scenario S1, requests for a non-existent model ID returned a 400 error from Amazon Bedrock, and the call did not proceed to the next destination based on the default trigger. This aligns with the API Reference sentence in Section 4.1: 4xx is not among the default triggers. In scenario S5, the 400 error from alternative destination 1 did not proceed to alternative destination 2, but was returned to the caller. This aligns with the statement in Section 4.5, which indicates that the alternative destination's triggerOn is not inherited. In the S5 records, reason was servedViaFallbackHop:1 even though the response returned to the caller was a 400 error. In this configuration, this reason records that the response from alternative destination 1 was returned, and does not indicate a successful outcome.

8.5 Timeouts and Streams

With a valid model on the primary and firstByteTimeoutMilliseconds set to 50 milliseconds (scenario S3), an error message was logged after the primary attempt in all three calls, and the alternative destination answered. Because the rule omits triggerOn, this article reads that the default trigger FIRST_BYTE_TIMEOUT fired once a time was set. In scenario S4, where the timeout was removed from the rule, only the primary was attempted in both calls, and the primary answered.

In scenario S6, which used the same rule as S3 and called InvokeModelWithResponseStream, an error message was logged for the primary attempt, followed by a streamed response from the alternative destination. The requestPath was /model/amazon.nova-micro-v1:0/invoke-with-response-stream. The calling SDK received the response as a standard Amazon Bedrock event stream. This configuration did not allow for simulating a failure of the destination after the first byte was sent to the caller, so the failure scenarios described in Section 6.2, which occur mid-stream, could not be tested.

8.6 Circuit Breaker

With the circuit breaker enabled, scenario S7 made 10 consecutive calls with the same rule as S2. In each call, the primary was attempted, and the alternative destination answered after two attempts. Prior to scenario S7, the same target and model combination had failed 11 times over approximately 4 minutes. In scenario S8, where the circuit breaker was disabled, all three calls also took two attempts. No log record showed that the circuit breaker skipped a destination.

Based on these observations, it is not possible to determine whether the circuit breaker does not count 4xx errors, or whether this number and interval of calls do not reach the threshold. This article does not estimate the circuit breaker threshold from this. Due to cost and the potential impact on the account, a large volume of requests designed to open the circuit breaker was not sent.

8.7 Metrics and What Was Not Tried

In the AWS/Bedrock-AgentCore namespace, the CloudWatch metrics that have this gateway as a dimension were listed 3 minutes, 6 minutes, and about 80 minutes after the calls. All three lists were empty. In this setup, metrics related to calls to the inference endpoint did not appear in this namespace. Whether they appear in another namespace or under other dimensions was not checked, so this article does not state that there are no metrics.

The following items were not tested within this configuration:

Destination 5XX and 429 responses, UNROUTABLE, unreachable destinations, and CONNECTION_ERROR.
Alternative destinations on a translated route, and requests that specify a guardrail.
How retries in the caller's SDK overlap the gateway chain. The SDK retries were fixed to a single attempt.
Attributes within trace spans. To avoid changing the account's CloudWatch Transaction Search setting, traces were not delivered.

9. Discrepancies and Omissions in the Documents

This section lists instances where the documents differ in their wording, as well as information that could not be found in the documents. This article does not presume to identify any of these as errors.

9.1 Tie-Breaking

When multiple targets match the same model with the same strength, and no specific rule exists, the DG connector and provider pages describe the tie-breaking process as follows. Both pages begin with the same rule: if an Amazon Bedrock target is available, use it.

The DG connector page is written as follows:

When multiple targets match the same model at the same specificity, the gateway defaults
to the Amazon Bedrock target if one is among the matches. Otherwise, it selects one of the
matching targets at random on each request, so requests for the same model can land on
different targets.

The DG provider page includes the following second sentence:

Otherwise, it distributes requests across the matching targets in round-robin order.

The two pages are the pages for connector targets and for provider targets. This article presents this as a difference between the pages by target type. No statement was found on which tie-breaking process would apply when both the connector target and the provider target match the same model. This is the no statement was found state. To pin requests to a target, both pages say to prefix the model ID with the target name, and both also mention rewriting the model field in a request interceptor. That a rule can pin the target is stated on the DG fallback page (Section 3.2).

9.2 Words Related to Triggers

The What's New announcement (October 22, 2026) gives, as examples, the following:

along with the conditions that trigger a switch, such as throttling, server errors, or
response timeouts.

The default trigger for the API Reference is 5XX and three other errors, excluding 429. There is also no timeout unless a timeout period is specified. This article reads the "throttling" and "response timeouts" of the announcement as examples of conditions that can be selected when specifying a triggerOn value and timeout periods, not as a description of the default behavior.

The DG fallback page and the release notes use similar words. The DG fallback page opens with the first sentence below, and the release notes describe the feature with the second.

a request can fail when the target that serves the requested model is throttled, out of
capacity, or unreachable.

so that when the primary destination is throttled, out of capacity, or unreachable, the
gateway automatically tries the next target or model that you specified.

Neither sentence lists the default triggers, and the release notes continue with You configure which failure types trigger fallback. This article reads the "throttled" of these two sentences in the same way as the announcement: a destination's 429 triggers a fallback only when triggerOn lists it (Section 4.3).

9.3 Retry Logic and Fallback Logic

The What's New announcement and the DG fallback page describe in different words what the caller no longer needs.

so your agents stay available without each client implementing its own retry logic.

As described in Section 7.4, the DG fallback page words this as follows:

without each calling client implementing its own fallback logic.

This article treats the DG wording as the specification. The responsibility for retrying the last error and any failures that occur mid-stream remains with the caller, as outlined on the DG InvokeModel page and the DG fallback page.

9.4 The Last Error and the Last Response

The DG refers to what the caller receives when all destinations fail as "the last error," while the API Reference describes it as "the last response." This article reads these two terms as different ways of describing the same event. A response that matched a trigger is an error response, so the last response is an error. A response that matches no trigger is returned directly during the chain (see Section 3.3).

9.5 Paths Subject to Token Rate Limits

The DG Rate limit best practices page states that token rate limits apply to the following paths:

Supported paths: Token rate limits apply only to requests on known inference paths
(/v1/chat/completions, /v1/messages, /v1/responses). Requests to other paths are not
subject to token limits.

However, the DG InvokeModel page states that token-based rate limits also apply to InvokeModel requests.

Amazon Bedrock AgentCore Policy and token-based rate limiting apply to InvokeModel
requests as they do to other inference traffic.

The path for InvokeModel is /inference/model/{model}/invoke, and it is not included in the three paths listed on the Rate limit best practices page. This is the state the sources disagree. This article does not definitively state that either page is incorrect. When applying token-based rate limits to InvokeModel requests, it is necessary to cross-reference both pages to confirm.

9.6 Region

The What's New announcement specifies the scope of services as follows:

These features are available in all AWS Regions where Amazon Bedrock AgentCore Gateway
is available (except those in China).

This article does not list the number of Regions or provide a complete list. For the scope of availability, this article defers to the What's New announcement and the AgentCore Regions page.

9.7 What Was Not Found

The following items were not found in the sources in Section 1.3. This is the no statement was found state. This article does not use terms like "unavailable" or "impossible" to describe these items.

Numerical values for the conditions under which a circuit opens, the duration it remains open, the percentage of traffic allowed through in a half-open state, and the location where the state is maintained (Section 5.7).
Whether codes like 429 are counted by the circuit breaker (Section 5.4).
Regarding rules that do not include timeout specifications, when requests to a stopped destination will conclude (Section 4.4).
How gateway-generated 400 codes are handled within a chain (Section 4.6).
The format of errors that occur mid-stream (Section 6.2).
The field names in routing logs, the types of logs, and dedicated fallback metrics (Section 7.2).
Whether the span attribute aws.agentcore.gateway.resolved_target_name refers to the target used after a fallback (Section 7.2).
A statement that SDK retries and gateway chains overlap (Section 7.4).
The relationship between session stickiness and fallback mechanisms (Section 3.4).
Whether Converse and ConverseStream are accepted at inference endpoints. The tables on the DG InvokeModel page, the What's New announcement, and the release notes do not list Converse as a supported API. This is the state the documentation does not list.
The behavior when cross-Region inference profiles, Amazon Bedrock's Intelligent Prompt Routing, and gateway fallback mechanisms are combined. (Section 8 used a cross-Region inference profile as an alternative destination, and it answered; that is an observation, not a statement in the sources.)

10. Failure Modes

This section lists potential failures, categorized by symptoms, causes, and methods of verification.

Failing to fall back on 429 errors: The default trigger does not include 429. Ensure that triggerOn includes 429 or 4XX, and also include the default 5XX status codes, along with the three errors, in the same triggerOn configuration. Even when configured, the gateway's own throttling will not trigger a fallback.
triggerOn configured, but no longer falling back on 5xx errors: triggerOn is replacing the default behavior. Add 5XX and any other necessary error codes to the triggerOn configuration.
Failing to fall back on stopped destinations: firstByteTimeoutMilliseconds is not configured. This article reads that, if it is not configured, no first-byte timeout is enforced. For rules configured with triggerOn, ensure that errors includes FIRST_BYTE_TIMEOUT.
Failing to proceed to alternative destination 2 due to 429 errors or timeouts on alternative destination 1: triggerOn and timeout configurations are not inherited by alternative destinations. Configure each alternative destination individually.
Attempts to reach the alternative destination result in 400 errors: The schema of the payload in the alternative destination's model does not match the request body. The gateway does not perform schema transformations. This also occurs when an InvokeModel request with guardrails is sent to the translation path. According to the DG InvokeModel page, a 400 error also occurs when an InvokeModel request without contentType is sent to the translation path.
The response is cut off mid-stream: After sending the first byte to the caller, the gateway will not fall back. The caller must retry.
Unable to determine which model responded: There is no indicator in the response. Verify using routing logs.
The gateway makes more calls than expected for a single invocation: Each attempt is counted. This article reads that, if the SDK retries the last error, the gateway will retrace the chain. Disabling the circuit breaker will cause a failing primary destination to be attempted every time.
matchModels matches more models than expected: If the operator is omitted, it defaults to StringLike, where asterisks (*) in the value act as wildcards. To match the value exactly, use StringEquals.
Rules are being rejected: One cause is that the same target and model combination is written twice in the rule; the primary and alternative destinations must be defined as separate combinations. The DG also requires at least one targetFallback entry in a fallback block and at least one non-blank entry in a matchModels condition.

11. Frequently Asked Questions About Model Fallback in AgentCore Gateway

Q1. If a `fallback` is specified in the rules, will it switch to a different model due to throttling?

By default, not on a destination's 429. The default trigger conditions are 5XX status codes, as well as CONNECTION_ERROR, FIRST_BYTE_TIMEOUT, and UNROUTABLE errors. A 429 status code is not included in the default trigger.

If you want to switch to a different model when encountering a 429 or 4XX status code on the destination, you need to specify 429 or 4XX in the triggerOn setting's statusCodes. Since triggerOn overrides the default settings, if you also want to switch models on 5XX status codes and the three error conditions, you must include those as well in the same triggerOn setting. The gateway's own throttling will not trigger a switch to a fallback model.

Q2. On a timeout, does the request switch to a different model by default?

It does if the timeout occurs during the connection stage. This is because a connection timeout falls under CONNECTION_ERROR, which is included in the default trigger. However, if the destination becomes unresponsive after a connection is established, this article reads that it will not switch models unless firstByteTimeoutMilliseconds is configured.

Q3. Does the value specified in `triggerOn` get added to the default value?

No, it does not. The API Reference states that when you specify a value in triggerOn, only the status codes and errors you write will trigger a fallback. If you require the default 5XX status codes and the three error conditions, you must also specify those.

Q4. Do the `triggerOn` setting and timeout specified in the primary configuration also apply to alternative destinations?

No, they do not. The API Reference page for FallbackTarget states that neither setting is inherited from the route. If you omit triggerOn in the alternative destination, the default trigger is used. If you omit the timeout setting, there is no timeout.

Q5. How many failures occur before the circuit breaker skips a destination?

No specific numerical values were found in the documentation. The API Reference only states that targets and models will be skipped if repeated failures occur. No information was found regarding how long the circuit stays open or the share of traffic allowed during a half-open state. The only setting available is to enable or disable the circuit breaker functionality.

Q6. If the circuit breaker is open, will alternative destination 2 also be skipped?

The last remaining destination in a chain will not be skipped. The API Reference, in its description of circuit breakers, states that the gateway will always attempt the target remaining at the end of the chain. This does not mean that alternative destination 2 will always be attempted. If the previous destination returns a response that matches no trigger, alternative destination 2 will not be attempted.

Q7. If the destination becomes unavailable during a stream, will the gateway continue processing with a different model?

No. A fallback can only occur before the gateway sends the first byte to the caller. Once the stream has begun, the caller will receive a partial response and an error, and it is the caller that will attempt to retry.

Q8. Can you tell which model provided the response, based on the response itself?

No, not from anything the gateway adds. The DG states that the gateway does not include an indicator to the caller indicating which destination it used. You can verify the destination by checking the gateway's routing logs. However, the specific names of the fields in the routing logs were not found in the sources. In the setup in Section 8.3, the resolvedModelId and reason fields in body.routing showed which destination answered.

Q9. With the gateway in place, can the SDK retry mechanism be removed?

The documentation does not explicitly state whether the retry mechanism should or should not be removed. The DG InvokeModel page indicates that the InvokeModel client receives the last error and relies on the SDK's built-in retry functionality. This article reads that, if the SDK retry mechanism is retained, the gateway walks the chain again when the last error falls into the retryable error types, potentially resulting in a maximum of 3 × max_attempts attempts to the destination. If the SDK retry mechanism is removed, the application will be responsible for handling retries for both mid-stream failures and the last error.

Q10. Can fallback mechanisms also be used with the Converse API?

The documentation lists InvokeModel (including InvokeModelWithResponseStream), OpenAI's Chat Completions and Responses, and Anthropic's Messages as supported dialects for the inference endpoint. Converse and ConverseStream are not listed on the DG InvokeModel page, in the announcements, or in the release notes. No statement was found stating that they are unusable, nor any stating that they are usable.

12. Summary

Fallback behavior is determined by the fallback block in the gateway rules targeted at the inference target. After the primary destination, list one or two alternative destinations; the gateway will then check the circuit breaker for each destination and attempt them one by one.
The default trigger is 5XX errors and three other errors; 429 is excluded. triggerOn overrides the default setting. The gateway's own throttling does not trigger a fallback.
If firstByteTimeoutMilliseconds is omitted, there is no timeout. This article reads that FIRST_BYTE_TIMEOUT will not be triggered. triggerOn and timeout settings are not inherited by alternative destinations.
The circuit breaker is enabled by default and maintains state for each combination of target and model. The circuit breaker does not skip the last remaining destination. Information regarding the conditions for opening the circuit breaker and associated time values was not found in the sources.
Fallback only occurs before the first byte is sent to the caller. The caller handles failures during the stream and retries.
Only one response is returned to the caller, and there is no indication of which destination provided the response. Routing logs are the location to verify this. This article reads that the SDK retry mechanism and the gateway chain overlap by multiplication.

13. References

Amazon Bedrock AgentCore Developer Guide: Model routing and fallback
Amazon Bedrock AgentCore Developer Guide: Add rules to a gateway
Amazon Bedrock AgentCore Developer Guide: Configuration bundle propagation and observability
Amazon Bedrock AgentCore Developer Guide: Session stickiness for weighted rules
Amazon Bedrock AgentCore Developer Guide: Rate limit best practices
Amazon Bedrock AgentCore Developer Guide: Use the Amazon Bedrock InvokeModel API with inference targets
Amazon Bedrock AgentCore Developer Guide: Inference connector targets
Amazon Bedrock AgentCore Developer Guide: Inference provider targets
Amazon Bedrock AgentCore Developer Guide: Core concepts for Amazon Bedrock AgentCore Gateway
Amazon Bedrock AgentCore Developer Guide: AgentCore generated gateway observability data
Amazon Bedrock AgentCore Developer Guide: Quotas for Amazon Bedrock AgentCore
Amazon Bedrock AgentCore Developer Guide: Release notes for Amazon Bedrock AgentCore
Amazon Bedrock AgentCore Control API Reference: Fallback
Amazon Bedrock AgentCore Control API Reference: FallbackTarget
Amazon Bedrock AgentCore Control API Reference: FallbackTriggers
Amazon Bedrock AgentCore Control API Reference: CircuitBreakerConfiguration
Amazon Bedrock AgentCore Control API Reference: StaticRoute
Amazon Bedrock AgentCore Control API Reference: CreateGatewayRule
AWS CloudFormation Template Reference: AWS::BedrockAgentCore::GatewayRule
AWS CloudFormation Template Reference: AWS::BedrockAgentCore::GatewayRule Fallback
AWS CloudFormation Template Reference: AWS::BedrockAgentCore::GatewayRule FallbackTarget
AWS SDKs and Tools Reference Guide: Retry behavior
What's New: AgentCore Gateway now supports Bedrock InvokeModel APIs and model fallback
Claude Platform Docs: Refusals and fallback

References:
Tech Blog with curated related content

Written by Hidekazu Konishi