LLM Inference Resilience Patterns on AWS - Retries with Backoff and Jitter, Circuit Breakers, and Stream Recovery

First Published:
Last Updated:

1. Introduction

Large language model (LLM) calls behave differently from the CRUD calls most application code is built around. A call to Amazon Bedrock can be slow (a long generation is normal, not a bug), it can fail transiently (the service throttles you at peak, a model warms up, a connection drops mid-stream), and under sustained load it can jam — a burst of retries from every instance at once can turn a brief hiccup into a self-inflicted outage. If your client code treats an inference call like a local function that either returns instantly or never, the first busy afternoon in production will surface every one of these assumptions at the same time.

This article is a practical, implementation-level guide to the client-side resilience patterns that keep a generative AI application stable under those conditions: how to classify failures so you retry the right ones, how the AWS SDK's built-in retries (exponential backoff, jitter, and the retry quota) actually work and how to configure them, how to add an application-level circuit breaker on top, how to set timeout budgets that account for streaming, how to detect and recover from an interrupted stream, and how to degrade gracefully instead of failing hard. It is the design-pattern companion to the Amazon Bedrock Errors and Exceptions Reference on this site: that reference tells you what each exception means; this article tells you how to build the client so the exception rarely reaches your user.

The scope here is deliberately the client — the code that calls the model. Root-cause remedies for sustained throttling (Provisioned Throughput, cross-Region inference, prompt caching) belong to Amazon Bedrock Inference Throughput and Latency Optimization, and the delivery architecture for streaming responses to end users belongs to End-to-End Response Streaming Architecture for Generative AI on AWS. This article assumes those are separate concerns and focuses on the resilience layer that sits between your application and the model. For terminology, see the Amazon Bedrock Glossary.

The code examples use Amazon Bedrock through the AWS SDKs, and they are written to be model-agnostic: a MODEL_ID variable stands in for whichever current foundation model you invoke, because the resilience patterns are identical regardless of which model is behind the endpoint. Every configuration value, default, and API behavior below is drawn from the official AWS documentation; where a number depends on a setting or a version, that is called out rather than asserted as universal. The figure below is the map for the rest of the article: it shows the layers a request passes through and the point at which each mechanism acts.

LLM call resilience layers and where each mechanism acts, from the application request through the circuit breaker, the AWS SDK retry layer with its timeout budget, Amazon Bedrock, and the graceful-degradation fallback path

2. Classifying Failures: What to Retry and What Not To

Resilience starts with a single decision made correctly and consistently: is this failure worth retrying? Retrying a failure that can never succeed wastes latency and capacity; not retrying a failure that would have succeeded on the second try turns a blip into a user-visible error. The AWS SDKs already encode this decision, and understanding their taxonomy is the foundation for everything else in this article.

The SDK sorts every failed request into one of three buckets:
  • Transient errors — something momentary went wrong that is likely to succeed on a retry: a connection reset, a DNS hiccup, a socket timeout, an HTTP 500/502/503/504 with no more specific code, or a service-side InternalServerException. These are retried with a short base delay.
  • Throttling errors — the service actively rejected the request because you exceeded a rate or throughput limit: ThrottlingException, TooManyRequestsException, ProvisionedThroughputExceededException, and similar. These are retried too, but with a longer base delay, because the service is explicitly asking you to slow down.
  • Non-retryable errors — the request itself is wrong, and retrying it unchanged cannot succeed: ValidationException (malformed request), AccessDeniedException (missing IAM permission or expired credentials), ResourceNotFoundException (bad identifier). These are returned to your code immediately.

For Amazon Bedrock specifically, the mapping lines up as follows. The full catalog — every exception name, its HTTP status on each API surface, and the official fix — lives in the Amazon Bedrock Errors and Exceptions Reference; the table below is the resilience view of it: retry or not, and why.

* You can sort the table by clicking on the column name.
Exception (Bedrock)HTTPRetry?Why
ThrottlingException429Yes (throttling)Momentary rate/throughput limit; back off and retry
ModelNotReadyException429YesModel is warming up; the SDK retries it up to five times
ModelTimeoutException408Yes (transient)The request took too long; retry, and reduce work per request
InternalServerException500Yes (transient)Service-side error; retry with backoff
ServiceUnavailableException503Yes (transient)Temporary unavailability; retry, consider another Region
ModelStreamErrorException424YesStreaming-only; retry, handle inside the stream iteration
ValidationException400NoFix the request; retrying it unchanged cannot succeed
AccessDeniedException403NoFix IAM/credentials; not a transient condition
ResourceNotFoundException404NoFix the model or profile identifier
ServiceQuotaExceededException400No (special)A quota boundary, not a momentary rate limit; resubmit later or request an increase

Two distinctions on this table repay careful attention because they are the ones most often gotten wrong.

Throttling (429) is not the same as a quota boundary (400). ThrottlingException means you briefly exceeded your allowed rate — backing off and retrying with jitter is exactly right, and it usually succeeds within a few attempts. ServiceQuotaExceededException means you hit a hard quota; retrying in a tight loop cannot help, so the SDK does not treat it as automatically retryable. The correct response is to resubmit later or raise the quota. Conflating the two produces a client that hammers a quota boundary it can never get past. One API-surface nuance: Converse and ConverseStream do not list ServiceQuotaExceededException among their errors — they surface capacity limits as ThrottlingException; the quota exception appears on InvokeModel.

Some "failures" are not errors at all. A successful Converse response carries a stopReason, and a value of max_tokens means the model hit your maxTokens limit and returned a truncated answer — the call succeeded (HTTP 200). A guardrail that blocks or masks content also returns 200, with stopReason: guardrail_intervened. Neither raises an exception. If your resilience code retries on "the answer looks wrong," it will retry these normal responses forever. Handle them as response data, not as failures. If you call Claude directly on the Anthropic API rather than through Bedrock, the analogous error taxonomy is in the Anthropic Claude API Errors Reference; the retry/no-retry logic is the same, only the status codes and names differ.

Everything that follows builds on this classification. Retries and circuit breakers act only on the retryable categories; non-retryable errors should propagate immediately so a human or a calling service can fix the underlying request.

3. SDK Retries: Backoff, Jitter, and Retry Quotas

The single most effective resilience improvement for most Bedrock clients is not custom code — it is configuring the retry behavior the AWS SDK already ships with. The SDK implements exponential backoff, full jitter, and a retry quota that prevents retry storms, and it does so consistently across languages. This section covers how it works internally and how to configure it, because the defaults are not always what you want.

3.1 The Three Retry Modes

Every AWS SDK offers three retry modes, described in the AWS SDKs and Tools Reference Guide:
  • standard — the recommended default. Retries with exponential backoff and full jitter and includes a retry quota (a token bucket, covered below); under the updated 2026 retry behavior it additionally uses error-type-specific base delays (Section 3.2). Standardized across all SDKs, so a Python service and a JavaScript service behave the same way.
  • adaptive — everything in standard mode, plus a client-side rate limiter that watches for throttling responses and slows the client down. Crucially, adaptive mode can delay or block the initial request, not just retries. The rate limiter operates per SDK client instance, so throttling on one resource slows every request from that client. This is the right choice for a client that hammers a single resource and expects frequent throttling (a batch processor calling one model at high volume), and the wrong choice for a client that fans out across many resources or tenants. AWS documents adaptive mode as experimental and subject to change, and does not recommend it as a general default.
  • legacy — the pre-standard behavior each SDK used before standard mode existed. It has no standardized retry quota and its retry counts, backoff timing, and retryable error sets vary between languages. It exists for backward compatibility only.

The boto3 default is legacy, not standard. This is the resilience trap most Python Bedrock code falls into without knowing it. The cross-SDK reference lists standard as the default, but the AWS SDK for Python (boto3/botocore) still defaults to legacy for backward compatibility, with a default of 5 total attempts. If you never set retry_mode, you get legacy behavior — no standardized retry quota, and a different retryable-error set. Always set the mode explicitly. Several other SDKs (for example, the AWS SDK for JavaScript v3) default to standard, so a multi-language fleet can silently behave inconsistently unless you pin the mode everywhere.

3.2 Backoff and Jitter, Concretely

When the SDK decides to retry, it does not retry immediately — it waits, and the wait grows with each attempt. Under the updated retry behavior (opt-in today via AWS_NEW_RETRIES_2026=true, the default from November 2026 — see Section 3.3), the base delay additionally depends on the error type: transient errors use a short base (on the order of tens of milliseconds) because they usually clear in milliseconds, while throttling errors use a base around a second because the service has explicitly asked for room to recover. The pre-2026 default standard mode documents a single exponential backoff with a base factor of 2 and does not publish per-error-type base delays — so treat the split as a property of the opt-in/2026 behavior, not of every SDK you run today.

The delay is not a fixed exponential curve; it is randomized. The SDK computes each retry delay as, in effect:
delay = random(0, 1) x min(cap, base_delay x 2^attempt)

The 2^attempt term is the exponential growth, the min(cap, ...) term caps any single wait (the SDK caps individual backoff at 20 seconds), and the random(0, 1) multiplier is full jitter. Jitter is not cosmetic. Without it, every client that failed at the same instant — say, when a shared dependency blipped — would compute the same backoff and retry at the same instant, producing a synchronized wave of traffic (the "thundering herd") that re-triggers the failure. Full jitter spreads those retries uniformly across the backoff window so the service sees a steady trickle instead of a spike. The Amazon Builders' Library article "Timeouts, retries, and backoff with jitter" is the canonical treatment, and the companion post Exponential Backoff And Jitter compares full, equal, and decorrelated jitter variants. The practical takeaway is that jitter matters as much as backoff, and the SDK gives you both for free in standard and adaptive modes.

3.3 The Retry Quota: A Token Bucket That Prevents Retry Storms

Backoff and jitter smooth one client's retries over time. The retry quota solves a different problem: it stops a client from retrying at all when retries are clearly futile, so a widespread failure resolves faster instead of being prolonged by everyone's retry traffic.

Standard and adaptive modes maintain a token bucket per client. Each retry deducts tokens; a successful request restores tokens. When the bucket is empty, the SDK stops retrying and returns the error immediately — the request "fails fast" rather than waiting through backoff for a retry that will not help. During healthy operation the bucket stays full and has no effect; it only bites during a sustained failure, which is exactly when you want retries to stop. The quota never delays or blocks the initial request — only retries are gated. boto3's documentation calls this "circuit-breaking functionality," which is a useful mental model but should not be confused with the application-level circuit breaker in the next section: the SDK's token bucket is per client and counts retry tokens; an application circuit breaker tracks the health of a downstream dependency across whole calls and short-circuits new requests entirely.

Retry-quota costs are changing (default November 2026). AWS has announced an updated retry behavior for the SDKs and tools. You can opt in today with the environment variable AWS_NEW_RETRIES_2026=true, and it becomes the default in November 2026. The token-bucket capacity stays at 500, but the token cost per transient retry rises (in the pre-2026 standard mode a transient retry deducts 5 quota tokens; under the updated behavior it deducts 14, while a throttling retry deducts 5), and backoff timing and the DynamoDB defaults change. If you have explicitly set standard or adaptive, your mode choice does not change — how the mode behaves does. If you have set no retry configuration at all, the update goes further: for the SDKs that still default to legacy — boto3 and the AWS CLI among them — it also switches the default mode itself to standard, giving those clients a retry quota for the first time. That silent flip is one more reason to pin the mode explicitly (Section 3.1); an explicit legacy setting is left unchanged. Test on a non-production workload before the cutover. Because these exact token values differ between the current default and the updated behavior, treat the mechanism — "a token bucket that fails fast when retries are futile" — as the durable concept, and consult the official reference for the numbers that apply to your SDK version and opt-in state.

3.4 Configuring Retries in Code

Retry settings resolve by precedence, highest to lowest: explicit configuration in code, then the AWS_RETRY_MODE / AWS_MAX_ATTEMPTS environment variables, then the retry_mode / max_attempts keys in the shared config file (~/.aws/config), then the SDK default. A value set at a higher level overrides a lower one. The default max attempts under standard mode is 3 (one initial request plus two retries); setting it to 1 disables retries entirely.

There is one subtlety in boto3 that causes real bugs: what max_attempts counts depends on where you set it. In the shared config file or via AWS_MAX_ATTEMPTS, it is the total number of attempts including the initial request. In a botocore Config object, max_attempts is the number of retries (excluding the initial request). To avoid the ambiguity, use total_max_attempts in Config objects, which is always the total.

Python (boto3), setting the mode explicitly — the important fix — and pinning timeouts (covered in Section 5):
import boto3
from botocore.config import Config

# total_max_attempts is unambiguous: it always counts the initial request.
# Setting the mode explicitly avoids boto3's legacy default.
bedrock_config = Config(
    region_name="us-east-1",
    retries={
        "total_max_attempts": 4,   # 1 initial request + 3 retries
        "mode": "standard",        # "standard" | "adaptive" | "legacy"
    },
    connect_timeout=3.0,           # seconds; default is 60
    read_timeout=60.0,             # seconds; default is 60
)

client = boto3.client("bedrock-runtime", config=bedrock_config)

response = client.converse(
    modelId=MODEL_ID,              # a current foundation model ID
    messages=[{"role": "user", "content": [{"text": "Hello"}]}],
)

# Inspect how many retries the SDK actually performed:
attempts = response["ResponseMetadata"]["RetryAttempts"]
print(f"SDK retried {attempts} time(s) before succeeding")

The same settings via environment variables (useful for containers, where you want to tune retries without a code change):
export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=4   # total, including the initial request

The AWS SDK for JavaScript v3 exposes the same two knobs on the client constructor:
import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime";

const client = new BedrockRuntimeClient({
  region: "us-east-1",
  maxAttempts: 4,        // total attempts, including the initial request
  retryMode: "standard", // "standard" | "adaptive" | "legacy"
});

const response = await client.send(
  new ConverseCommand({
    modelId: MODEL_ID,
    messages: [{ role: "user", content: [{ text: "Hello" }] }],
  }),
);

3.5 Confirming Retries Are Happening

Two signals confirm your configuration is working. First, the response metadata: response["ResponseMetadata"]["RetryAttempts"] reports how many retries occurred before success. Second, debug logging — with boto3's logger at DEBUG, standard/adaptive mode emits messages from botocore.retries.standard, including the telling line Retry needed but retry quota reached, not retrying request, which is your direct evidence that the token bucket is engaging (and a strong hint that something downstream is broadly unhealthy). If you enabled retries but never see a retry in your logs under load, verify you did not silently land on legacy mode.

4. Circuit Breakers for LLM Calls

Retries handle the individual call: this request failed, wait and try again. A circuit breaker handles the dependency: Bedrock (or a specific model, or a specific Region) appears to be broadly unhealthy right now, so stop sending requests to it for a while instead of making every user wait through the full retry sequence before failing. The two are complementary — retries recover from blips within a call; the breaker protects the system when blips become a pattern.

There is no single managed "circuit breaker" service in AWS that wraps a Bedrock call; the circuit breaker is an application-level pattern you implement in your client (or adopt from a resilience library in your language). The SDK's retry quota, discussed above, is a narrower, per-client "fail fast when retries are futile" mechanism — a useful first line, but not a substitute for a breaker that reasons about a dependency's health across many calls and can short-circuit new requests before they are even attempted.

4.1 The State Machine

A circuit breaker is a small state machine with three states:
  • Closed — normal operation. Requests pass through to Bedrock. The breaker counts consecutive failures (or a failure rate over a window).
  • Open — the failure count crossed a threshold. The breaker "trips" and, for a cooldown period, rejects requests immediately without calling Bedrock at all. This is what protects you: instead of every request paying the full timeout-plus-retry cost during an outage, requests fail instantly and your degradation path (Section 7) takes over.
  • Half-open — after the cooldown elapses, the breaker lets a limited number of trial requests through. If they succeed, it concludes the dependency has recovered and returns to closed. If they fail, it returns to open and starts the cooldown again.

The transitions are: closed to open when failures exceed the threshold; open to half-open when the cooldown timer expires; half-open to closed on trial success; half-open to open on trial failure.

Circuit breaker state transitions for LLM calls: closed trips to open when failures reach the threshold, open moves to half-open when the cooldown elapses, and half-open returns to closed on a successful trial request or back to open on a failed one

4.2 Threshold Design for LLM Calls

Tuning a breaker for LLM traffic differs from tuning one for a fast CRUD API in three ways:
  • Count the right failures. Trip on retryable, dependency-level failures — throttling that survived the SDK's own retries, ServiceUnavailableException, InternalServerException, connection failures, timeouts. Do not trip on non-retryable client errors like ValidationException or AccessDeniedException: those reflect a bad request or a permissions problem, not an unhealthy dependency, and one malformed request should never open the circuit for everyone.
  • Set the cooldown to the timescale of real recovery. Throttling and capacity events resolve over seconds to low tens of seconds, so a recovery timeout in that range is a reasonable starting point — long enough to stop hammering, short enough to recover quickly. Tune against your observed recovery times.
  • Account for long, legitimate latencies. Because a slow generation is normal, a breaker that treats "slow" as "failed" will trip constantly. Let the timeout budget (Section 5) define failure; the breaker should react to timeouts and errors, not to the mere fact that a call took a while.

4.3 A Minimal Implementation

The following Python breaker wraps a Bedrock call. It is intentionally small and single-process; a fleet-wide breaker would share state (for example, in a low-latency store such as Amazon ElastiCache), but the state machine is identical. It uses a monotonic clock so it is immune to wall-clock adjustments, and it only counts dependency-level failures as trips.
import time
import threading
from botocore.exceptions import ClientError, EndpointConnectionError, ReadTimeoutError, ConnectTimeoutError

# Bedrock exceptions that indicate the dependency (not the request) is unhealthy.
RETRYABLE_ERROR_CODES = {
    "ThrottlingException",
    "ServiceUnavailableException",
    "InternalServerException",
    "ModelNotReadyException",
    "ModelTimeoutException",
    "ModelStreamErrorException",
}
TRANSPORT_FAILURES = (EndpointConnectionError, ReadTimeoutError, ConnectTimeoutError)


class CircuitOpenError(Exception):
    """Raised when the breaker is open and the call is short-circuited."""


class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=20.0, half_open_max_calls=1):
        self._failure_threshold = failure_threshold
        self._recovery_timeout = recovery_timeout
        self._half_open_max_calls = half_open_max_calls
        self._state = "closed"
        self._failures = 0
        self._opened_at = 0.0
        self._half_open_calls = 0
        self._lock = threading.Lock()

    def _is_dependency_failure(self, exc):
        if isinstance(exc, TRANSPORT_FAILURES):
            return True
        if isinstance(exc, ClientError):
            return exc.response["Error"]["Code"] in RETRYABLE_ERROR_CODES
        return False

    def call(self, func, *args, **kwargs):
        with self._lock:
            if self._state == "open":
                if time.monotonic() - self._opened_at >= self._recovery_timeout:
                    self._state = "half-open"       # cooldown elapsed: allow trial calls
                    self._half_open_calls = 0
                else:
                    raise CircuitOpenError("Bedrock circuit is open; failing fast")
            if self._state == "half-open" and self._half_open_calls >= self._half_open_max_calls:
                raise CircuitOpenError("Bedrock circuit is half-open; trial limit reached")
            if self._state == "half-open":
                self._half_open_calls += 1

        try:
            result = func(*args, **kwargs)
        except Exception as exc:
            if self._is_dependency_failure(exc):
                self._on_failure()
            raise
        else:
            self._on_success()
            return result

    def _on_success(self):
        with self._lock:
            self._failures = 0
            self._state = "closed"                  # any state -> closed on success

    def _on_failure(self):
        with self._lock:
            if self._state == "half-open":
                self._state = "open"                # trial failed: reopen and restart cooldown
                self._opened_at = time.monotonic()
                return
            self._failures += 1
            if self._failures >= self._failure_threshold:
                self._state = "open"
                self._opened_at = time.monotonic()


breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=20.0)

def invoke_with_breaker(client, **kwargs):
    return breaker.call(client.converse, **kwargs)

Note the division of labor: the SDK still owns per-call retries with backoff and jitter (you configured that in Section 3), so by the time an exception escapes client.converse, the SDK has already retried it and given up. The breaker then decides whether this escaped failure, combined with recent history, means the dependency is unhealthy enough to stop trying. Layering them this way — SDK retries inside, breaker outside — is what makes the system both forgiving of blips and protected against sustained failures.

5. Timeout Budgets

A timeout is the deadline that turns "slow" into "failed," which is what makes retries and breakers possible in the first place — without a timeout, a stuck call simply hangs forever and no amount of retry logic ever runs. The Amazon Builders' Library is blunt about it: set a timeout on every remote call. For LLM inference the challenge is that generations are legitimately slow, so the timeout has to be generous without being infinite.

Think in terms of three layers:
  • Connection (connect) timeout — how long to wait to establish the TCP/TLS connection. This should be short (a couple of seconds); a slow connect indicates a network or endpoint problem, not a slow generation, and there is no reason to wait long. In botocore this is connect_timeout (default 60 seconds — far too long for a connect; lower it).
  • Read (request/socket) timeout — how long to wait for the response once connected. For non-streaming inference this must accommodate the full generation time, so it is the timeout you size against your longest realistic prompt/response, not a default. In botocore this is read_timeout (default 60 seconds).
  • Overall (end-to-end) budget — the total time your caller is willing to wait, across connect, read, and any SDK retries. This is not a single SDK setting; it is a property of your application. Remember that SDK retries multiply wall-clock time: with retries and backoff, the total time a call can consume approaches timeout x max_attempts plus the accumulated backoff. If a user-facing request has a 30-second budget, a 60-second read timeout with three attempts can blow it several times over. Size the per-attempt timeout and max_attempts so their product fits the overall budget, and enforce the budget at the caller (for example, with an outer deadline that abandons the call and triggers degradation).

Concretely, the botocore Config shown in Section 3.4 sets connect_timeout=3.0 and read_timeout=60.0. For a client that only ever issues short completions you would lower read_timeout; for one that generates long documents you would raise it and correspondingly lower max_attempts so the overall budget stays bounded.

Streaming changes the read-timeout calculation in an important way. With ConverseStream or InvokeModelWithResponseStream, the read timeout applies between chunks, not to the whole response — the connection delivers tokens continuously, so as long as chunks keep arriving within the read timeout, a multi-minute stream does not time out. This is one of the operational reasons streaming is preferable for long generations: a non-streaming call must complete within one read-timeout window, whereas a stream only has to keep producing. It also means the failure you must design for shifts from "the whole call timed out" to "the stream went quiet or broke mid-flight," which is the subject of the next section.

6. Stream Recovery

Streaming is the right default for anything a user watches unfold, but it introduces a failure mode non-streaming calls do not have: the response can break after it has started. You have already delivered half an answer, and then the connection drops or the service emits a ModelStreamErrorException mid-stream. Recovering well here is what separates a polished experience from one that dumps a truncated paragraph and a stack trace on the user.

6.1 How Interruptions Surface

Bedrock's streaming operations can fail in two distinct places, and robust code handles both:
  • Before the stream opens — the same synchronous exceptions as a non-streaming call (ValidationException, AccessDeniedException, ThrottlingException, ResourceNotFoundException) are raised when you invoke the operation, before any chunk arrives. Treat these exactly as in Sections 2 to 4.
  • Mid-stream — once chunks are flowing, an error can arrive as an event inside the stream. ModelStreamErrorException is specific to streaming and carries the underlying model's original status code and message; InternalServerException, ValidationException, ThrottlingException, and ServiceUnavailableException can also appear as stream events. Because these surface while you are iterating over chunks, the try/except must wrap the iteration, not only the initial call.
import boto3
from botocore.exceptions import ClientError, EventStreamError

client = boto3.client("bedrock-runtime", config=bedrock_config)  # from Section 3.4

def stream_with_buffer(messages):
    """Yield text chunks while buffering everything received so far.
    On a mid-stream failure, the caller still has the partial text and the
    reason the stream ended."""
    buffered = []
    try:
        response = client.converse_stream(modelId=MODEL_ID, messages=messages)
        for event in response["stream"]:
            if "contentBlockDelta" in event:
                text = event["contentBlockDelta"]["delta"].get("text", "")
                buffered.append(text)
                yield text
            elif "messageStop" in event:
                # Normal completion. stopReason is data, not an error:
                # "end_turn", "max_tokens", "guardrail_intervened", ...
                return {"status": "complete",
                        "stop_reason": event["messageStop"].get("stopReason"),
                        "text": "".join(buffered)}
            elif ("internalServerException" in event
                  or "modelStreamErrorException" in event
                  or "validationException" in event
                  or "throttlingException" in event
                  or "serviceUnavailableException" in event):
                # Mid-stream error delivered as an in-band event; keep the partial.
                return {"status": "interrupted",
                        "error": next(iter(event)),
                        "text": "".join(buffered)}
    except (EventStreamError, ClientError) as exc:
        # Transport break or exception raised during iteration; keep the partial.
        return {"status": "interrupted",
                "error": str(exc),
                "text": "".join(buffered)}

Buffering as you stream is the foundational move: it means an interruption never loses what the user already saw, and it gives you the material to decide what to do next.

6.2 Resume Strategies, and Their Honest Limits

Once a stream breaks, there are three ways forward, and it is important to be candid about the trade-offs because the tempting option is also the riskiest.
  • Retry the whole request. Discard the partial output and re-issue the call from scratch. This is the safe default: because inference on a fixed prompt has no server-side side effects, re-issuing is idempotent in the sense that matters (see Section 8), and you get a clean, coherent answer. The cost is latency (the user waits again) and wasted work (you pay for the partial generation and the full one). For short responses this is almost always the right choice.
  • Continue from where it stopped. Send a follow-up request whose context includes what was generated so far and ask the model to continue. This avoids regenerating the beginning, but it is genuinely risky: Amazon Bedrock's streaming APIs do not expose a resume token or byte offset that lets the service continue an interrupted generation server-side — there is no "resume from chunk N." Any continuation is a new generation seeded with the partial text, so the model may repeat the last sentence, change tone or formatting at the seam, or diverge from the original trajectory. Coherence at the join is not guaranteed. Use this only where you can tolerate and, ideally, post-process the seam, and never present it as equivalent to an uninterrupted response.
  • Return the partial with an honest signal. For some UIs the best answer is to keep the buffered text, mark it visibly as incomplete, and offer the user a "continue" or "regenerate" action. This respects the user's time and avoids fabricating completeness.

The design rule is to match the strategy to the stakes: retry-from-scratch for short or high-stakes answers, partial-with-signal for long exploratory ones, and continue-from-context only with eyes open to the quality risk. Whichever you choose, buffer as you go so the choice is available. For the broader architecture of delivering streams to clients (WebSocket APIs, Lambda response streaming), see End-to-End Response Streaming Architecture for Generative AI on AWS.

7. Graceful Degradation and Fallbacks

When retries are exhausted and the circuit is open, the last question is: what does the user get? Failing with an error is one answer, but often a worse one than a degraded-but-useful response. Graceful degradation is the practice of having a planned, ordered set of fallbacks so a Bedrock outage becomes a quality reduction rather than a hard failure.

A fallback chain for an LLM feature typically has these rungs, from best to last-resort:
  • A fallback model. If the primary model is throttled or unavailable, route to an alternate — a different model, or the same model family in another Region reached through a cross-Region setup. This is the highest-fidelity fallback because the user still gets a generated answer. The throughput mechanisms that reduce the need to fail over (Provisioned Throughput, cross-Region inference) are covered in Amazon Bedrock Inference Throughput and Latency Optimization.
  • A cached response. For queries that recur, a previously computed answer (exact-match or semantic cache) is instant and needs no model call at all. Caching also cuts load in normal operation; on token- and prompt-level efficiency see Anthropic Claude API Prompt Caching and Token Efficiency.
  • A non-generative response. When no model answer is available, fall back to something deterministic: a canned answer, results from a plain keyword search over your knowledge base, a template, or a clear message that the AI feature is temporarily degraded with a path to a human. This is the floor, and every production LLM feature should have one.
def answer_with_degradation(messages, cache):
    # 1. Try the primary model through the circuit breaker (Sections 3-4).
    try:
        return invoke_with_breaker(client, modelId=PRIMARY_MODEL_ID, messages=messages)
    except (CircuitOpenError, ClientError):
        pass
    # 2. Try a fallback model (fresh breaker or direct call).
    try:
        return client.converse(modelId=FALLBACK_MODEL_ID, messages=messages)
    except ClientError:
        pass
    # 3. Serve a cached answer if we have one for this query.
    cached = cache.get(cache_key(messages))
    if cached is not None:
        return cached
    # 4. Non-generative floor: never leave the user with nothing.
    return {"degraded": True,
            "text": "Our AI assistant is temporarily unavailable. "
                    "Here are the most relevant help articles instead."}

Two cautions. First, do not silently degrade in a way that hides a problem: a degraded response should be observable (a metric, a flag on the response) so operators know the primary path is failing even while users are shielded. Second, keep the fallback simpler and more robust than the primary — a fallback that depends on the same throttled model, or that is itself fragile, is not a fallback.

8. Idempotency and Deduplication

Every retry raises the same question: if the first attempt actually reached the service and did its work before the response was lost, does retrying cause that work to happen twice? For remote calls with side effects, a timeout does not tell you whether the side effect occurred — only that you did not hear back. This is why the Builders' Library article Making retries safe with idempotent APIs treats idempotency as a prerequisite for safe retries, not an afterthought.

For LLM inference the news is mostly good, with two caveats:
  • The inference call itself is effectively idempotent. InvokeModel, Converse, and their streaming variants are read-like with respect to your resources: they generate text from your prompt and change no state you own. Retrying a failed inference call is safe in the sense that it will not corrupt data or double-charge a resource you manage. What you may get is a different answer, since generation is not deterministic — fine for most uses, worth noting if you cache or compare outputs.
  • The surrounding workflow is where duplication bites. The risk is not the model call; it is everything wrapped around it. If your handler calls Bedrock, then writes the result to a database or sends it to a user, and a retry (at the SDK, your own code, or an upstream gateway like API Gateway or a message queue with at-least-once delivery) re-runs the whole handler, you can generate and deliver two answers. The model call was safe; the write and the send were not.

The remedy is a deduplication key carried end to end. Assign each logical request a stable identifier at the edge (a client-supplied request ID, or one you mint on first receipt), and make the state-changing steps conditional on it: a conditional write keyed on the request ID (so the second attempt is a no-op), or a short-lived record in a low-latency store that marks "this request ID already produced and delivered an answer." Where an upstream AWS service offers an explicit idempotency mechanism — a client request token, message deduplication — use it rather than reinventing one. The principle is to push idempotency to the boundary where the side effect happens, so that however many times the inner call is retried, the user-visible effect occurs once.

9. Detecting Retry Storms

The failure mode that makes resilience engineering worth doing is the self-inflicted one: a brief upstream blip causes every client to retry, the retries themselves become the load, and the service cannot recover because the retry traffic never lets up. This is the retry storm (and its cousin, the thundering herd — many clients retrying in lockstep). The patterns in this article are largely designed to prevent it — jitter de-synchronizes retries, the retry quota makes clients give up when retries are futile, and the circuit breaker stops traffic to an unhealthy dependency — but you still need to see it happening.

The signals that a retry storm is forming or underway:
  • A rising ratio of attempts to successes. If total request attempts climb while successful responses stay flat or fall, clients are retrying into a wall. ResponseMetadata.RetryAttempts aggregated across your fleet is a direct measure; a sudden rise is the earliest warning.
  • Throttling rate climbing after an initial error spike. A short burst of ThrottlingException is normal; a sustained and growing throttling rate after some other failure often means retries are now the cause, not the symptom.
  • The retry quota engaging. The Retry needed but retry quota reached, not retrying request debug line, or the equivalent metric, means the SDK's own guard is firing — individual clients have concluded retries are futile. Seeing this across many clients at once is a clear storm signal.
  • Circuit breakers tripping en masse. If your application breakers are opening across instances simultaneously, the dependency is broadly unhealthy and your fleet has (correctly) stopped hammering it.

Building the dashboards, metrics, and alarms that surface these signals is an observability concern in its own right, and it is covered in LLMOps Observability and Evaluation Architecture on AWS rather than repeated here. The resilience-specific point is what to do when you see a storm: the fastest mitigation is almost always to reduce retry aggression fleet-wide (lower max_attempts, confirm standard/adaptive mode with its quota is actually engaged, and make sure jitter is on), let the circuit breakers hold traffic back, and lean on the degradation path (Section 7) to keep users served while the dependency recovers. Increasing retries during a storm makes it worse; the counterintuitive correct move under load is to try less, not more.

10. Frequently Asked Questions

Should I write my own retry-with-backoff loop, or use the SDK's?

Use the SDK's. Standard and adaptive modes already implement exponential backoff, full jitter, error-type-specific base delays, and a retry quota, consistently across languages. A hand-rolled loop usually omits jitter or the quota and often double-retries (your loop plus the SDK's). Configure the SDK (Section 3.4) rather than reimplementing it; reserve custom code for the layers the SDK does not provide, such as the application circuit breaker and the degradation chain.

Why does my Python (boto3) client behave differently from my JavaScript client?

Most likely because boto3 defaults to legacy retry mode (5 attempts, no standardized quota) while the AWS SDK for JavaScript v3 defaults to standard. Set the mode explicitly in every SDK — mode: "standard" in a botocore Config, retryMode: "standard" on the JS client, or AWS_RETRY_MODE=standard in the environment — so a multi-language fleet behaves consistently.

Is a circuit breaker a built-in Amazon Bedrock feature?

No. Bedrock and the AWS SDKs give you retries with backoff, jitter, and a retry quota, but the circuit breaker — the closed/open/half-open state machine that stops traffic to an unhealthy dependency — is an application-level pattern you implement in your client or adopt from a resilience library. boto3 describes its retry quota as "circuit-breaking," but that is a narrower per-client fail-fast mechanism, not a full breaker.

Can I resume an interrupted Bedrock stream from where it broke?

Not server-side. Bedrock's streaming APIs do not expose a resume token or offset, so there is no way to have the service continue the same generation. Your options are to retry the whole request (safe, but the user waits again and you may get a different answer), to seed a new request with the partial text and ask the model to continue (avoids regeneration but risks repetition or a tonal seam), or to keep the buffered partial and let the user choose. Buffer as you stream so all three options remain open.

Which errors should trip my circuit breaker?

Only dependency-level, retryable failures that survived the SDK's own retries: throttling, ServiceUnavailableException, InternalServerException, ModelNotReadyException, timeouts, and connection failures. Do not trip on non-retryable client errors such as ValidationException or AccessDeniedException — those are bad requests or permissions problems, and letting one open the circuit would deny service to everyone over a single caller's mistake.

How do I keep retries from turning a small outage into a big one?

Rely on the mechanisms designed for it: full jitter (on by default in standard/adaptive mode) de-synchronizes retries, the retry quota makes clients stop when retries are futile, and a circuit breaker halts traffic to an unhealthy dependency. Cap max_attempts so the worst-case wall-clock time stays within your budget, and when a storm is already underway, reduce retry aggression rather than increasing it.

11. Summary

Resilient LLM inference on AWS is built in layers, each solving a problem the layer beneath it cannot:
  • Classify failures first. Retry transient and throttling errors; never retry ValidationException, AccessDeniedException, or ResourceNotFoundException unchanged; and treat stopReason values and guardrail interventions as normal 200 responses, not failures.
  • Configure the SDK's retries rather than rewriting them. Standard (or adaptive) mode gives you exponential backoff, full jitter, and a retry-quota token bucket for free. Set the mode explicitly — especially in boto3, which defaults to legacy — and size max_attempts against your latency budget. Watch for the updated retry behavior that becomes the default in November 2026.
  • Add a circuit breaker above the SDK. The closed/open/half-open state machine stops traffic to a broadly unhealthy dependency so requests fail fast into your degradation path instead of each paying the full retry cost. Trip it only on dependency-level failures.
  • Set timeout budgets that respect streaming. Short connect timeout, a read timeout sized to the generation (per-chunk for streams), and an overall caller deadline that accounts for retries multiplying wall-clock time.
  • Recover streams honestly. Buffer as you stream; prefer a clean retry-from-scratch for short answers; treat "continue from context" as a quality risk, not a free resume; and be candid that Bedrock offers no server-side resume.
  • Degrade gracefully and stay idempotent. Have an ordered fallback chain (fallback model, cache, non-generative floor), make it observable, and push deduplication to the boundary where side effects happen so retries never double-deliver.
  • Watch for retry storms. Rising attempt-to-success ratios, sustained throttling, the retry quota engaging, and breakers tripping en masse are your signals — and the correct response under load is to try less, not more.

Together these turn the three inherent properties of LLM calls — slow, occasionally failing, prone to jamming under load — from production incidents into handled, observable, and largely invisible events.

Related reading on this site:

12. References

Retry behavior (AWS SDKs and Tools Reference Guide)
Retries (AWS SDK for Python / Boto3 documentation)
Config Reference (Botocore documentation)
Retries in the AWS CLI (AWS Command Line Interface User Guide)
Announcing updated retry behavior for AWS SDKs and Tools (AWS Developer Tools Blog)
Timeouts, retries, and backoff with jitter (Amazon Builders' Library)
Making retries safe with idempotent APIs (Amazon Builders' Library)
Exponential Backoff And Jitter (AWS Architecture Blog)
ConverseStream (Amazon Bedrock Runtime API Reference)
InvokeModelWithResponseStream (Amazon Bedrock Runtime API Reference)
Implement retry logic and exponential backoff for Amazon Bedrock (AWS re:Post)

References:
Tech Blog with curated related content

Written by Hidekazu Konishi