Event-Driven Architecture Anti-Patterns on AWS - Failure Modes, Root Causes, and How to Design Around Them

First Published: 2026-07-20
Last Updated: 2026-07-20

1. Introduction

Event-driven architectures fail in ways that are quiet until they are catastrophic. A single misconfigured trigger can turn a normal upload into a runaway loop; an unstated assumption about ordering can corrupt a ledger; a dead-letter queue that nobody watches can silently swallow a day of orders. None of these failures announce themselves at deploy time. They surface under load, at the boundary between services, in exactly the places that are hardest to test.

This article is a catalog of the recurring anti-patterns that cause those failures on AWS, and — more importantly — the official guidance that tells you how to avoid each one. It is deliberately the back side of a companion article. Where Event-Driven Serverless Architecture on AWS shows how to assemble API Gateway, Lambda, EventBridge, SQS, SNS, Step Functions, and DynamoDB into one resilient system, this article isolates the specific mistakes that break such systems and grounds every fix in an AWS warning, a Well-Architected best practice, or an Amazon Builders' Library article.

Two rules govern everything below. First, every anti-pattern is anchored to official AWS guidance — a documentation Warning or Note, a Well-Architected best practice, an Amazon Builders' Library article, or an AWS blog with explicit "avoid this" guidance. This is not a collection of war stories. Second, there are no invented incidents. Each failure is described as a mechanism — "given this configuration, the service's documented behavior propagates like this" — derived from the specification, not from anecdote. Where AWS has shipped a guardrail, I say exactly what it does and, just as importantly, what it does not cover.

This is written for engineers who already run event-driven systems on AWS and want to know why they break and what to avoid before they do. If you are still choosing between SQS, SNS, and EventBridge, start with the decision guide; if you are assembling the reference architecture, start with the companion build article. Read this one as the pre-mortem: the eight failure modes below are the ones most likely to reach production unnoticed, ordered so the widest-blast-radius runaways come first.

Service behavior in this article was verified against official AWS documentation as of 2026-07-18. Message-delivery semantics, quotas, and guardrail coverage change over time; treat the dated statements as a snapshot and re-check the linked sources before you rely on them.

2. How to Read This Catalog

Every entry follows the same three-part shape, and reading them in that order is the point:

Anti-Pattern — what the mistake looks like in a real design, and what happens at runtime.
Why It Happens — the specification-level reason the failure propagates. This is where the AWS documentation is cited: the failure is a consequence of documented behavior, not a bug.
How to Avoid or Fix — the official recommendation, plus a See also link to the companion cluster where the correct implementation lives in full.

Two axes help you triage which entries matter most for your system. Likelihood is how easily the mistake slips into a design that otherwise looks correct — some of these are defaults you have to actively override. Blast radius is how far the damage spreads before something stops it: a single corrupted item, one starved consumer, or an account-wide runaway. An anti-pattern that is both easy to commit and wide in blast radius is where to spend your first review cycle.

A note on scope. This is the failure catalog; it is not a re-explanation of how to build the architecture correctly. Each fix points to the companion articles for the full pattern — the reference architecture, the messaging decision guide, the concurrency guide — rather than repeating them here. When a fix says "give every consumer its own queue," the why and the how live in Event-Driven Serverless Architecture on AWS; this article's job is to name the mistake and cite the guardrail.

Catalog at a glance:

#	Anti-Pattern	Trigger it hides in	Primary official control
3	Infinite Event Loops	A function writes to the resource that triggers it	Recursive loop detection; separate input/output resources
4	Assuming Order Where None Exists	Treating standard queues/buses as ordered	FIFO where order is required; best-effort ordering docs
5	Missing Idempotency at Consumers	Trusting at-least-once to mean exactly-once	Idempotent consumers; conditional writes
6	Retry Storms and Visibility Timeout Mismatch	Visibility timeout shorter than processing time	Visibility ≥ 6× function timeout; backoff with jitter
7	Poison Messages without Partial Batch Response	One bad record fails a whole batch	`ReportBatchItemFailures`; DLQ
8	DLQ as a Black Hole	A DLQ with no alarm and no redrive plan	Alarm on DLQ depth; redrive to source
9	Schema Evolution without Compatibility	Producers change payloads with no contract	Schema registry; producer and consumer validation
10	Fan-Out without Backpressure	Producers outrun consumers and downstreams	Maximum concurrency; fail fast and limit queues

3. Infinite Event Loops

Anti-Pattern. A function writes its output back to the resource that triggers it. The classic shape is an object handler wired to an S3 bucket that writes its derived object — a thumbnail, a normalized copy — back into the same bucket. The write is a new ObjectCreated event, which invokes the function again, which writes again. The same pattern appears when a consumer re-publishes to the same SNS topic or SQS queue it reads from, or when an EventBridge rule matches an event that one of its own targets emits. Invocations multiply without bound until throttling, a guardrail, or a billing alarm intervenes — and the cost is incurred whether or not the loop is caught quickly.

Two runaway failure modes in event-driven systems: an infinite event loop where a function's output re-triggers itself, and a retry storm where a visibility-timeout mismatch and un-jittered retries multiply load on a slow dependency — Two runaway failure modes in event-driven systems: an infinite event loop where a function's output re-triggers itself (left), and a retry storm where a visibility-timeout mismatch and un-jittered retries multiply load on a slow dependency (right).

Why It Happens. The trigger source and the write target are the same resource, so the effect of one invocation becomes the cause of the next. AWS documents this directly for S3: "If your Lambda function uses the same bucket that triggers it, it could cause the function to run in a loop. For example, if the bucket triggers a function each time an object is uploaded, and the function uploads an object to the bucket, then the function indirectly triggers itself." Lambda ships a guardrail — recursive loop detection, enabled by default — that uses an X-Ray "lineage" trace primitive to count how many times a function is invoked by the same triggering event; after roughly 16 invocations it stops the next one and emits a RecursiveInvocationsDropped CloudWatch metric (synchronous callers get a RecursiveInvocationException; asynchronous events go to the function's DLQ or on-failure destination). The critical limitation is what it covers: recursive loop detection works only between Lambda, Amazon SQS, Amazon S3, and Amazon SNS. As the documentation states, "When another AWS service such as Amazon DynamoDB forms part of the loop, Lambda can't currently detect and stop it." A loop that routes through EventBridge or DynamoDB is not caught — the most dangerous gap, because teams assume the guardrail is always watching.

How to Avoid or Fix. Separate input and output: write derived objects to a different bucket, or scope the trigger to a prefix that the function never writes to. When output must land in the same resource, break the cycle with a marker the trigger filters out — object metadata, an event attribute, or a DynamoDB flag the function checks before acting — the pattern AWS recommends in its guidance on avoiding recursive invocation with S3 and Lambda. Because the built-in guardrail does not cover EventBridge or DynamoDB loops, add the backstop AWS explicitly recommends: CloudWatch alarms on unusual concurrency or invocation spikes, plus a billing alarm or AWS Cost Anomaly Detection so a loop through an uncovered service cannot run up an unbounded bill unnoticed. Keep recursive loop detection on unless you have a deliberate recursive workload, in which case scope the opt-out to individual functions with PutFunctionRecursionConfig rather than disabling the guardrail account-wide.

Leading signal. A RecursiveInvocationsDropped value greater than zero is a definitive indicator, but it lags; the earlier warning is a sudden spike in a function's ConcurrentExecutions or Invocations with no matching increase in legitimate traffic. Alarm on both.

See also: the synchronous-to-asynchronous boundary and precise event routing that keep producers and consumers separated in Event-Driven Serverless Architecture on AWS, and the feature history of the fan-out primitive in AWS History and Timeline regarding Amazon SNS.

Official sources: Process Amazon S3 event notifications with Lambda (recursion Warning), Use Lambda recursive loop detection to prevent infinite loops, Avoiding recursive invocation with Amazon S3 and AWS Lambda.

4. Assuming Order Where None Exists

Anti-Pattern. The design treats a standard queue, a standard topic, or an event bus as if it delivered events in the order they were produced. A consumer applies "balance +100" then "balance ×2" assuming that sequence; a state machine assumes "created" always arrives before "updated." Most of the time the events do arrive in order, so the assumption survives testing and low traffic. Then, under load, two events are reordered, the consumer applies them in the wrong sequence, and the resulting state is silently wrong — the hardest class of failure to detect, because nothing errors.

Why It Happens. Ordering is a guarantee you must opt into, and the default services do not provide it. Amazon SQS standard queues offer best-effort ordering: as the documentation states, "occasionally, messages might be delivered in an order different from which they were sent." Amazon EventBridge is explicit that it "does not guarantee event order will be maintained." Neither is a defect — both trade ordering for throughput and availability, and both say so. Reordering is not rare enough to design around by hoping; it is a documented, expected behavior that appears precisely when volume rises.

How to Avoid or Fix. First, decide whether you actually need ordering — most event flows only need ordering within a key (one account, one order), not globally. Where you do need it, use a service that guarantees it: SQS FIFO queues preserve strict order within a MessageGroupId, and SNS FIFO topics do the same for fan-out. Be aware of the cost model that ordering imposes: for FIFO event source mappings, Lambda concurrency is capped by the number of message group IDs (or the maximum concurrency setting, whichever is lower), so a single group ID serializes processing. Where you cannot or should not adopt FIFO, remove the dependency on order instead of assuming it — carry a monotonic version or timestamp in the payload and have the consumer reject or reconcile out-of-order updates (a conditional write that only applies if the incoming version is newer). Ordering assumed is a latent bug; ordering enforced by a version check is a correctness property.

Leading signal. There is no queue metric for "processed out of order," which is exactly why it is dangerous. The practical detector is an application-level invariant: a monotonic version or sequence number that lets a consumer notice a gap or a backwards step and reject it, turning a silent corruption into a caught error.

See also: the deep trade-offs between standard and FIFO, and when each is the right routing primitive, in AWS Messaging and Event Routing Decision Guide; the ordering timeline for the queue itself in AWS History and Timeline regarding Amazon SQS.

Official sources: Amazon SQS at-least-once delivery and best-effort ordering, Event-Based Processing for Asynchronous Communication (AWS Architecture Blog), Delivery level for AWS service events, Exactly-once processing in Amazon SQS.

5. Missing Idempotency at Consumers

Anti-Pattern. A consumer performs a non-idempotent side effect — charge a card, decrement inventory, send an email, insert a row — exactly once per message, assuming each message is delivered exactly once. It is not. The same message is eventually delivered twice, and the side effect happens twice: a double charge, negative stock, a duplicate notification. Like reordering, this is invisible at low volume and emerges at scale, which is what makes it so common in systems that "worked in staging."

Why It Happens. At-least-once delivery is the default contract across the event-driven stack, and duplicates are a documented consequence of it, not an anomaly. Amazon SQS standard queues store copies of each message across multiple servers; if a server holding a copy is unavailable during a delete, "you might get that message copy again when you receive messages," and the documentation's instruction is unambiguous: "Design your applications to be idempotent (they should not be affected adversely when processing the same message more than once)." Visibility timeout does not close the gap — the SDK guides state plainly that "when using standard queues, visibility timeout isn't a guarantee against receiving a message twice." EventBridge carries the same contract: at-least-once delivery, "meaning duplicate messages can be introduced." Duplicates are structural. A consumer that is not idempotent is not merely fragile; it is incorrect by the services' own definitions.

How to Avoid or Fix. Make the effect idempotent, not just the code. Choose a stable business idempotency key — an order ID, not the SQS message ID, because the same business event can arrive under different message IDs — and enforce it at the point of effect. For create-once semantics, a DynamoDB conditional write with attribute_not_exists makes the second attempt a no-op. For general handler-level idempotency, the Powertools for AWS Lambda idempotency utility persists a result keyed by your idempotency key with an in-progress lock and a TTL, so a retried invocation returns the first result instead of repeating the work. Do not lean on FIFO "exactly-once" as an end-to-end guarantee: SQS FIFO deduplicates only within a 5-minute deduplication interval, and only against the producer's MessageDeduplicationId — the documentation notes that if an acknowledgement is lost and the message is resent with the same ID after that interval, "Amazon SQS can't detect duplicate messages." Consumer-side idempotency is the durable defense; FIFO dedup is a narrow, producer-side, time-boxed complement.

A minimal create-once guard makes the second delivery a no-op:

import boto3
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("orders")
try:
    # The second delivery of the same order_id fails the condition and is ignored.
    table.put_item(
        Item={"order_id": order_id, "status": "ACCEPTED"},
        ConditionExpression="attribute_not_exists(order_id)",
    )
except ClientError as e:
    if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
        return  # already processed - safe no-op
    raise

Leading signal. Like reordering, duplicate processing has no built-in queue metric; the detector is a uniqueness constraint at the point of effect (a conditional write that fails on the second attempt) or the hit rate of your idempotency store. A rising rate of "already processed" short-circuits confirms that duplicates are arriving and being neutralized correctly.

See also: conditional-write and Powertools idempotency implementations, and the idempotency-window pitfalls, in Event-Driven Serverless Architecture on AWS.

Official sources: Amazon SQS at-least-once delivery, Amazon SQS visibility timeout, Event-Based Processing for Asynchronous Communication (AWS Architecture Blog), Exactly-once processing in Amazon SQS.

6. Retry Storms and Visibility Timeout Mismatch

Anti-Pattern. A queue's visibility timeout is set shorter than the time the consumer actually needs to process a message. The consumer is still working when the timeout expires, SQS makes the message visible again, and a second consumer picks it up — so the message is processed twice and the first attempt's work is wasted. Under load this compounds: every slow message is redelivered mid-flight, in-flight duplicates pile up, maxReceiveCount is exhausted, and messages that would have succeeded are pushed to the DLQ. A related amplifier is naive retries against a slow dependency: each failure triggers an immediate retry, retries multiply the offered load, and a slowdown becomes an outage — a retry storm. This is the right-hand panel of the propagation diagram in section 3.

Why It Happens. Visibility timeout is the window in which a received message is hidden from other consumers; if the consumer does not delete the message within that window, SQS assumes failure and re-exposes it. When the window is shorter than real processing time, redelivery is not an error condition — it is the documented behavior, firing on every message that takes longer than expected. The retry-storm half is described in the AWS Well-Architected Framework's reliability guidance on controlling and limiting retries: when many clients retry failed operations as soon as an error occurs, the retried requests compete with new requests for bandwidth and capacity, multiplying the offered load into a retry storm that further reduces the availability of the already-degraded service. Simple backoff alone is insufficient, because clients that all back off by the same schedule re-collide in synchronized waves; the fix requires randomized (jittered) delays.

How to Avoid or Fix. Size the visibility timeout to the real work. AWS's guidance for SQS event sources is concrete: keep the SQS queue visibility timeout to at least six times the Lambda function timeout, plus the value of MaximumBatchingWindowInSeconds, which leaves room for Lambda to retry a failed batch before the message is redelivered. For jobs whose duration varies, extend the timeout while processing with ChangeMessageVisibility (a heartbeat), remembering the 12-hour ceiling from first receipt. Against the retry storm, apply the Builders' Library and Well-Architected controls together: exponential backoff with jitter, a capped maximum number of retries (REL05-BP03, "Control and limit retry calls"), and a circuit breaker that stops hammering a dependency that is already failing. These are not competing tactics — a correctly sized visibility timeout prevents self-inflicted duplication, and bounded, jittered retries prevent the load multiplication that turns one slow dependency into a system-wide failure.

Leading signal. Watch ApproximateAgeOfOldestMessage climbing while NumberOfMessagesReceived far exceeds successful deletions, Lambda Duration approaching the configured timeout, and DLQ depth rising — together these say messages are being redelivered faster than they are being completed.

See also: the worked symptom → root-cause → remediation walkthrough for a visibility-timeout incident, and the DLQ redrive that recovers from it, in Event-Driven Serverless Architecture on AWS; scaling and concurrency controls in AWS Lambda Concurrency and Scaling Guide.

Official sources: Amazon SQS visibility timeout, Configuring an Amazon SQS queue as an event source (visibility timeout guidance), Amazon Builders' Library: Timeouts, retries, and backoff with jitter, Well-Architected REL05-BP03: Control and limit retry calls.

7. Poison Messages without Partial Batch Response

Anti-Pattern. A Lambda function reads a batch of messages from an SQS queue and returns an error if any message in the batch fails. Because a batch either fully succeeds or fully fails, the whole batch is retried — including the messages that already processed cleanly. One malformed "poison" message therefore forces its healthy batch-mates to be reprocessed again and again, multiplying side effects and wasting compute, until the entire batch ages out to the DLQ. A single bad record degrades a whole slice of throughput.

Why It Happens. The default contract for an SQS event source mapping is all-or-nothing at the batch level: if the function returns an error, none of the batch's messages are deleted, so all of them — good and bad — return to the queue after the visibility timeout. Without a mechanism to tell Lambda which records failed, the service cannot distinguish the one poison message from its neighbors. AWS documents the mechanism that fixes this and the failure mode it addresses: a "poison pill" is a message "that is received but can't be processed," and left unmanaged it can even distort the ApproximateAgeOfOldestMessage metric and cause false alarms.

How to Avoid or Fix. Enable partial batch response. With ReportBatchItemFailures set on the event source mapping, the handler returns a batchItemFailures list containing only the message IDs it failed to process; Lambda then makes only those messages visible again and deletes the rest, so a healthy message is never reprocessed because of a poisoned neighbor. Pair it with a DLQ and a maxReceiveCount redrive policy so a message that genuinely cannot be processed is captured after a bounded number of attempts instead of circulating forever — AWS's guidance on capturing problematic messages is explicit that a DLQ "decreases the number of messages and reduces the possibility of exposing you to poison pill messages." For stream sources (Kinesis, DynamoDB Streams) the analogous controls are BisectBatchOnFunctionError, which splits a failing batch to isolate the bad record, together with MaximumRetryAttempts and an on-failure destination.

The handler shape that enables it returns the failed IDs and nothing else:

def lambda_handler(event, context):
    batch_item_failures = []
    for record in event["Records"]:
        try:
            process(record)              # your per-message work
        except Exception:
            # Report only the failed message; the rest are deleted from the queue.
            batch_item_failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": batch_item_failures}

Leading signal. The same message IDs recur in the DLQ, throughput drops without a matching rise in traffic, and — as AWS notes — a poison pill can distort ApproximateAgeOfOldestMessage into false alarms. Alarm on DLQ arrivals and sample the recurring records to identify the offending payload.

See also: partial batch response and the four dead-letter mechanisms disambiguated in Event-Driven Serverless Architecture on AWS; point-to-point source-to-target patterns that reshape batching in Amazon EventBridge Pipes and Event-Driven Architecture Patterns.

Official sources: Handling errors for an SQS event source in Lambda, Capturing problematic messages in Amazon SQS.

8. DLQ as a Black Hole

Anti-Pattern. A dead-letter queue is configured and then forgotten. Messages that fail processing land in it correctly — and stay there. No alarm fires on the DLQ's depth, no one inspects the messages, and there is no plan or tooling to reprocess them after the underlying bug is fixed. The DLQ becomes a black hole: it looks like resilience on the architecture diagram, but in practice it is a place where failed work goes to be quietly lost, often discovered only when a customer asks why their order never completed. The DLQ did its job; the operational loop around it was never closed.

Dead-letter queue as a black hole versus a closed operational loop: without an alarm and redrive the messages are silently lost, while an alarm on DLQ depth, investigation, and redrive to the source queue recover the messages after the fix — Dead-letter queue as a black hole (left) versus a closed operational loop (right): without an alarm and redrive the failed messages are silently lost; with an alarm on DLQ depth, investigation, and redrive to the source queue after the fix, the messages are recovered.

Why It Happens. A DLQ is only the first half of a message's failure lifecycle. AWS describes it as the mechanism that moves messages from a source queue to a dead-letter queue after maxReceiveCount is exceeded — it isolates unprocessable messages so they stop blocking the main queue. But isolation is not resolution. The documentation is explicit that once messages are in a DLQ you are expected to act on them: "examine logs for exceptions," "analyze the contents of messages," and "move messages out of the dead-letter queue using dead-letter queue redrive." A DLQ with no alarm and no redrive plan satisfies the letter of the pattern while defeating its purpose, because nothing in the system will tell you the DLQ is filling, and nothing will bring the messages back.

How to Avoid or Fix. Treat the DLQ as an incident signal, not a landfill. Configure a CloudWatch alarm on the DLQ's ApproximateNumberOfMessagesVisible — for a dead-letter queue, any value greater than zero is an incident worth investigating — which AWS documents as the recommended way to be alerted to messages arriving in a DLQ. Then close the loop: once the root cause is fixed and the consumer has recovered, use SQS DLQ redrive to move the messages back to their source queue (you can inspect a sample first and tune the redrive velocity), the second half of the lifecycle AWS added specifically so recovery no longer requires custom tooling. Two guardrails matter: a DLQ must be in the same account and Region as its source queue, and a FIFO queue requires a FIFO dead-letter queue — and AWS warns not to use a DLQ with a FIFO queue at all when preserving exact order through failures matters, because redriving reorders.

Leading signal. Here the absence of a signal is the failure mode: with no alarm, nothing tells you the DLQ is filling. The fix and the detector are the same action — alarm on the DLQ's ApproximateNumberOfMessagesVisible so the black hole announces itself the moment the first message lands.

See also: replay and reprocessing strategy, including EventBridge archive/replay alongside SQS redrive, in Event-Driven Serverless Architecture on AWS.

Official sources: Using dead-letter queues in Amazon SQS, Introducing Amazon SQS dead-letter queue redrive to source queues.

9. Schema Evolution without Compatibility

Anti-Pattern. A producer changes the shape of the events it emits — renames a field, changes a type, drops an attribute, nests what used to be flat — and consumers that were parsing the old shape break. Because producers and consumers are decoupled by design, the producer's team ships the change without knowing who depends on the old contract, and the failure shows up downstream: a consumer throws on a missing field, silently reads null where it expected a value, or routes on an attribute that no longer exists. In a fan-out topology, one payload change can fault several independent consumers at once.

Why It Happens. The decoupling that makes event-driven systems flexible also removes the compile-time contract that would have caught the break. Nothing in the transport enforces that today's event matches yesterday's schema. AWS's guidance on event validation names the mechanism plainly: "Producers may change data format over time, data may become corrupt, or interfaces between the producer and consumer may alter it," and failing to validate "can lead to data inconsistencies, downstream errors in processing and unnecessary costs." Without an explicit, versioned contract, every consumer is implicitly coupled to the exact byte shape the producer happens to emit today — a coupling that is invisible until it breaks.

How to Avoid or Fix. Make the schema an explicit, versioned artifact and validate on both sides. The Amazon EventBridge Schema Registry stores event schemas (in OpenAPI or JSONSchema form) and can discover them automatically — when schema discovery is on, EventBridge creates a new schema version whenever an event's structure changes, giving you a record of the evolution. Generate code bindings from the registry so consumers deserialize into a typed object and a breaking change surfaces at build time rather than in production. Follow the discipline AWS recommends: producers validate events before publishing, and consumers validate on receipt even from trusted sources, because the interface can change underneath them. Above all, evolve compatibly — add optional fields rather than renaming or removing required ones, and version the event type when a genuinely breaking change is unavoidable so old and new consumers can coexist during migration.

Leading signal. A spike in consumer Errors or deserialization failures immediately after a producer deployment is the classic fingerprint; if you validate against a registered schema, the validation-failure count is the direct signal. Correlate consumer error rates with producer release times to catch it fast.

See also: the event envelope and schema-evolution handling within the reference architecture in Event-Driven Serverless Architecture on AWS; routing precision on event attributes in AWS History and Timeline regarding Amazon EventBridge.

Official sources: Automating event validation with Amazon EventBridge Schema Discovery, How EventBridge retries delivering events (retry policy and DLQ).

10. Fan-Out without Backpressure

Anti-Pattern. A high-throughput producer fans out to consumers that call a downstream with a hard scaling limit — a relational database, a third-party API with a rate cap, a legacy service — and nothing throttles the consumers to that limit. Lambda scales out to drain the backlog as fast as it can, every concurrent invocation hits the constrained dependency at once, and the dependency is overwhelmed: connections exhausted, timeouts, cascading errors that then feed back into the queue as retries. The buffer that was supposed to protect the downstream instead becomes a pressure hose aimed at it. Worse, one uncapped queue can consume the account's entire Lambda concurrency pool and starve every other function in the account.

Why It Happens. Buffering decouples arrival rate from processing rate, but decoupling is not the same as limiting. By default, an SQS event source mapping can scale Lambda up to the account concurrency quota (1,000 by default, and an SQS event source can scale to 1,250 concurrent invocations), which is almost always more concurrency than a constrained downstream can absorb. The governing limit that matters is the consumer's rate, not the producer's — but nothing enforces that unless you configure it. The Amazon Builders' Library frames the underlying failure: an unbounded queue lets "message debt" accumulate into a backlog that drives up processing time until "work can be completed too late for the results to be useful," and the remedy is to fail fast and bound the queue rather than let it grow without limit.

How to Avoid or Fix. Put a ceiling on consumer concurrency so it matches what the downstream can take. For SQS event sources, set maximum concurrency on the event source mapping — an event-source-level cap that, in AWS's words, "can help prevent overwhelming downstream systems" and stops one queue from consuming all of the function's reserved concurrency or the account's quota. Keep the relationship between the two settings correct: maximum concurrency must not exceed the function's reserved concurrency, or Lambda may throttle the very messages you meant to admit. Complement the cap with the Well-Architected controls for distributed interactions — throttle requests (REL05-BP02) and "fail fast and limit queues" (REL05-BP04) so the system sheds or defers load instead of letting a backlog grow into stale, useless work. The design rule is simple: choose the concurrency ceiling from the downstream's capacity, and let the queue absorb the difference.

Leading signal. The downstream shows it first — timeouts, connection-pool exhaustion, and rising error rates on the constrained dependency — followed by Lambda Throttles and an SQS backlog (ApproximateAgeOfOldestMessage) that grows even while concurrency stays high. That combination means consumers are outrunning what the downstream can absorb.

See also: reserved vs. provisioned vs. maximum concurrency and event-source scaling in AWS Lambda Concurrency and Scaling Guide; bounding large parallel fan-out inside a workflow in AWS Step Functions Distributed Map Guide.

Official sources: Configuring scaling behavior for SQS event source mappings (maximum concurrency), Introducing faster polling scale-up for AWS Lambda functions configured with Amazon SQS (AWS Compute Blog), Amazon Builders' Library: Avoiding insurmountable queue backlogs, Well-Architected REL05-BP04: Fail fast and limit queues.

11. Cross-Cutting Design Principles

Read as a set, the eight anti-patterns reduce to a few principles that, applied consistently, prevent most of them at once.

Design for at-least-once, engineer for exactly-once effects. Duplicates and reordering are documented behaviors of the default services, not edge cases. Idempotent consumers (a stable business key plus a conditional write or the Powertools idempotency utility) and order enforced by a version check turn an at-least-once, best-effort transport into correct behavior. This single principle neutralizes anti-patterns 4 and 5 and softens 6.
Make failure paths first-class, not afterthoughts. A DLQ is only useful with an alarm on its depth and a redrive plan; a batch consumer needs partial batch response; a retrying client needs bounded, jittered backoff. Failure handling that is designed in — with defined destinations and a recovery path — is the difference between a five-minute redrive and a silent data-loss incident (anti-patterns 6, 7, and 8).
Bound everything that can grow. Loops, queues, retries, and concurrency all fail the same way: unbounded growth. Recursive loop detection, maxReceiveCount, a capped retry count, and maximum concurrency are the four limits that keep a local problem from becoming an account-wide one (anti-patterns 3, 6, and 10).
Make contracts explicit. The coupling that decoupled systems try to avoid reappears as an implicit dependency on event shape and delivery order. A versioned schema and an explicit ordering key convert invisible coupling into a checked contract (anti-patterns 4 and 9).
Observe the leading signals. Every failure here has an early indicator — RecursiveInvocationsDropped, ApproximateAgeOfOldestMessage, DLQ depth, Throttles, IteratorAge. Alarming on the signals that lead an incident, rather than the ones that trail it, is what turns these anti-patterns from outages into alerts.

The through-line is that none of these fixes are exotic. They are documented settings and standard patterns; the anti-pattern is almost always the default left unexamined.

A pre-deployment review checklist

Run these eight questions over any event-driven change before it ships; each maps to one anti-pattern above:

Loops: does any function write to a resource that can trigger it, and if so, is recursive loop detection plus a concurrency or billing alarm in place — especially for the EventBridge or DynamoDB paths the guardrail does not cover?
Ordering: does any consumer assume event order, and if it truly needs order, is it on FIFO or protected by a version check?
Idempotency: does every non-idempotent side effect have a stable business key enforced by a conditional write or an idempotency store?
Visibility timeout: is the SQS visibility timeout at least six times the function timeout plus the batching window, and are retries bounded and jittered?
Partial batch: is ReportBatchItemFailures enabled so one poison message cannot punish a whole batch?
Dead-letter: does every DLQ have an alarm on its depth and a tested redrive path?
Schema: is the event schema versioned and validated on both sides, and does the change add optional fields rather than break required ones?
Backpressure: is consumer concurrency capped to what the slowest downstream can absorb?

If any answer is "no" or "not sure," the section above that owns the question names the official control that closes the gap.

12. Frequently Asked Questions

12.1 Does Lambda's recursive loop detection mean I don't have to worry about infinite loops?

No. It is a valuable default, but it only detects loops between Lambda, SQS, S3, and SNS, and only stops them after roughly 16 invocations — so cost is still incurred before it fires. Loops that route through EventBridge or DynamoDB are not detected at all. Keep it enabled, but still separate input and output resources and add CloudWatch concurrency and billing alarms as a backstop.

12.2 Is a FIFO queue enough to guarantee no duplicate processing?

Not end to end. FIFO deduplication is producer-side and time-boxed: it removes duplicates within a 5-minute deduplication interval based on the MessageDeduplicationId. If the same message is resent after that window, or a duplicate originates somewhere other than the producer's send, FIFO will not catch it. Consumer-side idempotency (a stable business key and a conditional write) is the durable guarantee; FIFO dedup complements it but does not replace it.

12.3 How should I set the visibility timeout for a Lambda SQS consumer?

AWS's Well-Architected guidance is to set the SQS visibility timeout to at least six times the Lambda function timeout, plus the MaximumBatchingWindowInSeconds. That leaves room for Lambda to retry a failed batch before the message becomes visible to another consumer, which prevents the mid-flight redelivery that causes duplicate processing and DLQ pressure.

12.4 What is the difference between reserved concurrency and maximum concurrency for SQS?

Reserved concurrency guarantees a function a slice of the account's concurrency (and caps it there). Maximum concurrency is an event-source-level setting that limits how many concurrent invocations a particular SQS event source can drive — the right tool for protecting a downstream. They compose, but maximum concurrency must not be set higher than the function's reserved concurrency, or Lambda may throttle messages.

12.5 Where do failed messages and events actually go?

It depends on which hop failed, which is why the "four dead-letter mechanisms" matter: an SQS message a consumer cannot process goes to the SQS DLQ via the redrive policy after maxReceiveCount; an event EventBridge cannot deliver goes to the rule's DLQ; an asynchronous Lambda invocation that exhausts retries goes to the function's on-failure destination (preferred) or async DLQ. Whichever it is, alarm on it and have a redrive path. The full disambiguation is in the companion reference architecture.

12.6 How do I evolve an event's schema without breaking consumers?

Treat the schema as a versioned contract. Register it (the EventBridge Schema Registry can discover and version schemas automatically), generate code bindings so breakage surfaces at build time, validate on both producer and consumer, and evolve compatibly — add optional fields rather than renaming or removing required ones, and version the event type for genuinely breaking changes so old and new consumers can coexist during migration.

13. Summary

Event-driven systems on AWS do not usually fail because a service broke; they fail because a documented behavior was assumed away. Standard queues and buses are at-least-once and best-effort ordered by design — so duplicates and reordering are not bugs to be surprised by but contracts to engineer against with idempotent consumers and explicit ordering keys. A function wired to the resource that triggers it will loop, and the built-in guardrail does not cover every path. A visibility timeout shorter than real processing time redelivers healthy work; a batch that fails as a unit punishes its healthy messages; a DLQ with no alarm and no redrive silently loses them; a producer that changes payloads without a versioned contract breaks its consumers; and a fan-out with no concurrency ceiling turns a buffer into a pressure hose on a fragile downstream.

Every one of these has an official answer — an S3 Warning, a Well-Architected best practice, an Amazon Builders' Library article, a documented setting like ReportBatchItemFailures or maximum concurrency. The anti-pattern is almost never a missing feature; it is a default left unexamined. Design for at-least-once, make failure paths first-class, bound everything that can grow, make contracts explicit, and alarm on the leading signals — and the catalog above turns from a list of outages into a checklist you have already passed. For the positive counterpart — how to assemble these services into one resilient system end to end — continue with Event-Driven Serverless Architecture on AWS, and for how to choose the routing primitives in the first place, AWS Messaging and Event Routing Decision Guide.

hidekazu-konishi.com