Change Data Capture on AWS Beyond Zero-ETL - Logical Decoding, Streaming Connectors, and the Guarantees Each Capture Point Gives You

First Published: 2026-09-05
Last Updated: 2026-09-05

There is a common requirement: to direct changes from your production database somewhere other than your analytical platform. Perhaps you need to keep a search index synchronized, invalidate a cache, notify another microservice, or trigger downstream event-driven processes. The destination may vary, but the underlying need is the same: for someone other than the entity making the write to be notified when a write occurs.

AWS often already provides a solution for this. If your destination is Amazon Redshift or Amazon OpenSearch Service, zero-ETL integration options are available. If you need to migrate data from an older environment to a new one, AWS Database Migration Service is a suitable option. If either of these scenarios applies to you, you likely do not need to read further.

The challenge arises when these standard options do not fit your needs. Your specific combination of source and destination is not supported, or you need to transform values, apply conditional logic, or route data to multiple destinations in different ways. In these cases, you will need to design and implement a layer that extracts and transports the changes yourself. This is the subject of this article.

The first thing to understand about this layer is that the choice is not about selecting a specific tool. The key decision is where to place the capture point, and it is a decision about what you ask the source system to handle, rather than what you ultimately want to achieve. The same underlying event — a slowdown in processing — can manifest as storage exhaustion on a PostgreSQL database, or as the loss of necessary logs on a MySQL database. Two capture points configured for the same DynamoDB table might offer completely opposite guarantees regarding order and duplication.

This article will break down this layer, examining it from the perspective of each capture point. This is not a comparison of connector products. What you will take away from this article is a framework for making informed decisions about which capture point to choose, and a list of the operational responsibilities you will incur as a result of that choice. All information presented in this article was verified as of September 5, 2026.

1. Introduction - The Decisions This Article Supports
2. Where Zero-ETL and the Migration Tools Stop Being Enough
3. There Are Only Three Places to Read Changes From
4. The Replication Slot Is an Operational Object
5. Retention Fails in Opposite Directions
6. Separating Extraction from Delivery
7. Two Capture Points on the Same Table
8. Reading the Guarantee at the Capture Boundary
9. Where the Primary Sources Disagree
10. What This Path Takes Away
11. Failure Modes
12. Frequently Asked Questions
13. Summary
14. References

1. Introduction - The Decisions This Article Supports

1.1 The Situation This Article Assumes

This article is intended for architects designing systems that continuously stream changes from a production relational database or DynamoDB table to another service. This is not a project in transition; rather, it focuses on designing a system intended for ongoing, steady-state operation.

The situations this article has in mind are ones such as reflecting changes to an order table into a search index, or notifying multiple microservices of inventory updates. The destination may be an AWS analytics service, or it may require transformation even if it is an analytics service. Double-writing from the application code has been ruled out, because it introduces consistency problems. The remaining option is to read changes from the location where the database records its own modifications.

The most significant misunderstanding for readers in this situation is the belief that this choice is primarily about the destination. In reality, the decision is made on the source side. From the moment the system begins extracting data, the source database gains a new operational responsibility.

1.2 What Other Articles Already Cover

This article focuses specifically on the layer responsible for continuously extracting and moving changes. Other existing articles already cover related topics, so this article will not address them. The boundary is drawn first.

CDC as one phase of a migration - the seam between full load and CDC, the CDC start point, the two kinds of stop point, the difference between resume and restart, the DDL that CDC replicates, and the cutover - is covered by Heterogeneous Database Migration on AWS - Schema Conversion, Full Load with CDC, Data Validation, and a Cutover You Can Reverse. That article carries CDC in its title and gives its entire fourth section to that seam. This article does not write about migration. What it covers is the other case, the steady-state operation that never ends.

Continuous replication in a zero-ETL integration, enhanced logical replication, the boundary between the initial load and continuous replication, and history mode are covered by Zero-ETL Integrations on AWS - The Source and Target Matrix Across Amazon Redshift, AWS Glue, and Amazon OpenSearch Service. That article already holds the boundary with AWS DMS as well. This article starts from the point where you do not choose zero-ETL.

What happens after delivery - assumptions about ordering, idempotency on the consumer side, the decomposition of exactly-once, and backpressure - is covered by three articles. AWS Real-Time Streaming Data Pipeline Architecture Guide - Ingestion, Processing, and Delivery with Kinesis, Managed Service for Apache Flink, and OpenSearch treats backpressure, ordering, and exactly-once in its seventh section, and concludes with at-least-once delivery combined with idempotent effects. AWS Messaging and Event Routing Decision Guide - Choosing Between SQS, SNS, EventBridge, and Kinesis treats at-least-once as the default and the design of idempotency in its ninth section. Event-Driven Architecture Anti-Patterns on AWS - Failure Modes, Root Causes, and How to Design Around Them treats assuming order and missing consumer idempotency as failure modes. This article does not write about anything downstream of delivery. Section 8 names the segments and hands them over to these three.

The serverless implementation of the outbox pattern, meaning derived events generated through DynamoDB Streams and Amazon EventBridge Pipes, is covered by Event-Driven Serverless Architecture on AWS - Building Resilient Workflows with API Gateway, Lambda, EventBridge, Step Functions, and DynamoDB. This article does not write about implementing an outbox.

Job dependencies and scheduling, meaning orchestration and lineage, are covered by Data Pipeline Orchestration on AWS - What Carries Over, What the Runtime Makes You Rebuild, and Who Owns the Airflow Version. This article holds the layer that carries the changes, and does not write about the layer that schedules the work.

What happens once a slot blocks reclamation - dead tuples accumulating, heap and index bloat, XID wraparound, and diagnosis through postgres_get_av_diag() - is covered by PostgreSQL Autovacuum, Bloat, and Planner Statistics on Aurora and RDS - Why the Defaults Stop Being Enough, and What to Tune First. That article holds the downstream end of the causal chain, and this one holds the upstream end. This article does not write about the design of reclamation.

Logical replication as the switchover mechanism for a blue/green deployment is covered by Zero-Downtime Database Change on RDS and Aurora - Blue/Green Deployments, Upgrades, and Rollback Design. This article does not write about the switchover.

Isolation levels and anomalies are covered by Transaction Isolation on AWS Databases - Same Level Name, Different Anomalies, and the Statement That Changes Nothing. This article does not re-explain isolation levels.

How features arrived in Amazon DynamoDB over time is covered by AWS History and Timeline regarding Amazon DynamoDB - Overview, Functions, Features, Summary of Updates, and Introduction. This article does not build a timeline.

1.3 What Change Data Capture Means in This Article

The term "change data capture" (CDC) can refer to three distinct concepts, depending on the context. To settle what this article is about, they are distinguished here, at their first appearance.

Usage	Refers to	Treatment in This Article
A phase of a migration task	The stage that follows a full load, involving tracking changes in the source data. This phase has a defined end.	Not covered. Already covered in existing articles.
Internal mechanisms of managed integration	The mechanisms used internally within zero-ETL integration solutions to track changes in the source data. These mechanisms are not visible to the user.	Not covered. Already covered in existing articles.
Change stream as a product feature	A layer that continuously reads changes from a location where the database records its own changes and delivers them to other systems. This process is ongoing.	This article focuses on this.

From here on, CDC in this article means the third of these.

There is one more term this article settles up front. The place a change is read from is called a capture point. It names a place inside the database, not a tool. Whether using Debezium, AWS DMS, or custom scripts, if the same capture point is used, the same constraints apply. This article is structured around the concept of the capture point because these constraints are determined before the choice of tool.

1.4 Three Questions This Article Addresses

This article aims to answer the following three questions:

When neither zero-ETL nor a migration tool is used, what exactly do you end up building? There are three kinds of capture point. This section will clarify what each type reveals and what it conceals.
With that architecture, what will you need to continuously operate? Replication slots are not simply configurations; they are operational components. Furthermore, failures in maintaining them can lead to issues that manifest in completely opposite ways, depending on the system.
What is guaranteed at the boundaries of these capture points? When considering guarantees across different segments, this article only addresses the initial segment. The remaining segments are handed to the existing articles.

2. Where Zero-ETL and the Migration Tools Stop Being Enough

2.1 The Branch Conditions

Decisions about which layers to extract should always be a final consideration. There are two primary options to explore first.

First, determine if zero-ETL integration is feasible. If it is, the content of this article becomes largely unnecessary. While parameter configuration on the source side will still be required, extraction and delivery can be managed, and concerns related to slot operation, connector restart locations, and offset storage will disappear for the users.

Next, assess the feasibility of using AWS DMS. If the source and target combinations are supported, and the necessary transformations can be expressed within the capabilities of DMS transformation rules, this option will also be simpler to operate than building your own layer.

What is left after those two options usually falls under one of the following conditions.

Condition	Specifically
Destination is not supported	The destination is your own service or an external API, and is neither a search index nor an analytical platform.
Transformation is required	You need to perform value normalization, combine multiple tables, or supplement data with external sources during the extraction process.
Branching is necessary	You want to apply different transformations to the same changes, routing them to different destinations based on specific conditions.
Multiple destinations are required	You want to deliver extracted changes simultaneously to a search index, a cache, and an audit log.
You need to maintain control over delivery status	You want to manage and track how far the changes have been delivered.

⚠ The final condition should be approached with caution. Wanting to control it yourself is frequently a restatement of not having read what the managed path already guarantees. Only if, after carefully reviewing those guarantees, you still find them insufficient, should this condition apply.

2.2 What You Take On When You Cross That Line

Here is a list of the responsibilities that increase when you begin using this feature. If you find this list unacceptable, it may be worthwhile to re-evaluate your requirements.

Source database storage. Once you open the data stream, the source data is subject to the constraints of the system that retains the logs. See Sections 4 and 5 for details.
Consumption position. Someone must track how much data has been read. If this tracking is lost, you will need to restart from the initial snapshot. See Section 6.
Initial snapshot. Change streams only deliver changes. Data that already exists at the starting point must be transferred using an alternative method. See Section 6.
Schema changes. Depending on the data stream source, schema definition language (DDL) changes may or may not be visible. If not visible, downstream systems will need to independently detect column additions or deletions. See Sections 3 and 10.
Downstream resilience to duplicates and out-of-order data. The level of resilience required from downstream systems varies depending on the data stream source. See Sections 7 and 8.

3. There Are Only Three Places to Read Changes From

3.1 The Shape of the Choice

The candidates look numerous, but as long as you read what the database already writes for itself, they come down to three. While the number of tools available creates the illusion of a wide range of options, each tool ultimately reads from one of these three categories.

⚠ This article will only address these three categories. Methods for recording changes within the application itself — such as writing to an audit table via triggers, or writing event records within business transactions — are not included in these three categories. These approaches do not read what the database already records; they increase the number of writes, and the constraints they take on are entirely different. The existing articles named in Section 1.2 cover the latter.

Three capture points and what each one hands downstream

These three categories are as follows:

Capture Point	Applicable Systems	Data Being Read
Reading the write-ahead log (WAL) via logical decoding	Aurora PostgreSQL, RDS for PostgreSQL	The write-ahead log that the database writes for durability. Logical decoding restores changes at the row level.
Reading the binary log (binlog)	Aurora MySQL, RDS for MySQL, RDS for MariaDB	The change log that the database writes for replication.
Reading a native stream	DynamoDB Streams, Kinesis Data Streams for DynamoDB	A dedicated API that the database exposes for Change Data Capture (CDC).

These three categories are not equivalent. The first two leverage logs that the database writes for other purposes, while the third is specifically designed for CDC. This distinction is the source of most of the constraints that will be discussed in subsequent sections.

3.2 The Write-Ahead Log, Read Through Logical Decoding

PostgreSQL's Write-Ahead Log (WAL) is originally designed for crash recovery and replication. Because it records physical changes, it does not inherently provide information about which table and row have been modified, and in what way. Converting these changes into row-level logical changes is what logical decoding is all about.

In Aurora PostgreSQL and RDS for PostgreSQL, this feature is enabled through a parameter setting. The Aurora PostgreSQL user guide states the default value for rds.logical_replication as follows:

The default value for this parameter is 0, meaning that it's turned off by default.

Changing this parameter requires a restart. Furthermore, once enabled, additional WAL entries will begin to be written, even before any slots are created. The RDS for PostgreSQL user guide highlights this point as a note within the steps for enabling the feature.

These parameter changes can increase WAL generation, so set the rds.logical_replication
parameter only when you are using logical slots.

Therefore, it is not cost-effective to enable logical replication proactively, simply in anticipation of using it later. It is better to enable it when you actually need to consume the data. Because a restart is required, you need to factor that into your planning.

An output plugin decides the format that logical decoding emits. The choice of plugin determines what is required downstream, a topic covered in Section 6.2.

3.3 The Binary Log

MySQL-based binary logs are written to send changes to replicas. In a row-based format, something other than a replica can read it and still get the meaning. The essence of Change Data Capture (CDC) from the binary log is to connect as a client that speaks the MySQL replication protocol.

The most significant difference between WAL (Write-Ahead Logging) and the binary log lies in their default retention policies. Section 5 takes it up in full. However, it is important to understand that while both systems utilize logs originally written for different purposes, their retention designs are essentially reversed due to these differing original purposes.

3.4 The Native Stream

DynamoDB offers two dedicated mechanisms for reading changes: DynamoDB Streams and Kinesis Data Streams for DynamoDB. Both capture changes to individual items within a table, but they offer different guarantees. This is the most practical finding in this article, and Section 7 is given over to it entirely.

DynamoDB Streams possess three characteristics that the previous two systems lack.

First, the official documentation explicitly addresses the performance impact on the source. The DynamoDB developer guide states:

DynamoDB Streams operates asynchronously, so there is no performance impact on a table if you enable a stream.

Conversely, the WAL (Write-Ahead Log) system presents a different scenario. The previous section showed the opposite: the RDS user guide states that enabling the parameter can increase WAL generation. Furthermore, the AWS Database Blog notes that logical decoding can consume additional CPU and memory on the source instance. In essence, the cost of enabling data extraction is, in one case, officially denied, while in the other, it is officially acknowledged.

Second, the retention period is fixed or has an upper limit. Unlike slots, it does not accumulate indefinitely. Instead, if consumption lags, data is lost. This characteristic is similar to that of the binlog system discussed in Section 5.

Third, the concept of DDL (Data Definition Language) is largely absent. Because there is no schema, issues related to adding or removing columns do not arise. Instead, differences in attributes are reflected on a per-item basis downstream.

4. The Replication Slot Is an Operational Object

4.1 What a Slot Holds

A replication slot is an essential component when using logical decoding in PostgreSQL. While the name might suggest a configuration setting, it is actually an operational entity.

A slot essentially holds a consumption point. Debezium's official documentation describes this role as follows:

Debezium uses replication slots to stream changes from a database. Replication slots use a log
sequence number (LSN) as a persistent pointer to track the last position in the write-ahead
Log (WAL) that the connector has processed. This tracking mechanism enables PostgreSQL to
retain WAL segments until Debezium processes them.

The final sentence encapsulates the entirety of this section. The mere existence of a slot instructs the database not to discard any WAL (Write-Ahead Log) records that have not yet been read. As the consumer progresses and reads data, the slot's position advances, allowing older WAL records to be discarded. If the consumer stops, the position will not advance, and WAL records will continue to accumulate.

AWS also describes this in more direct terms. The RDS for PostgreSQL user guide states:

The logical replication slot has no information about the receiver of the stream. Also,
there's no requirement that the target be a replica database. If you set up a logical
replication slot and don't read from the slot, data can be written and quickly fill up your DB
instance's storage.

⚠ The initial sentence captures the essence of this mechanism. A slot does not know who its receiver is. It is unaware if the receiver has disappeared or become corrupted.

The only thing a slot knows is that the position has not advanced. It is the responsibility of the operations team to determine whether this represents a normal pause or an unrecoverable failure.

The Aurora PostgreSQL user guide describes this consequence as follows:

Leaving a logical replication slot inactive prevents the vacuum from removing obsolete tuples
from tables, so we recommend that you monitor replication slots and remove inactive slots as
needed.

⚠ What the official documentation states here is not simply an increase in storage space. It also means that reclamation itself stops. The potential consequences that follow — such as the accumulation of dead tuples, bloat, and the risk of XID (Transaction ID) wraparound — are discussed in the previously published article PostgreSQL Autovacuum, Bloat, and Planner Statistics on Aurora and RDS - Why the Defaults Stop Being Enough, and What to Tune First. This article focuses on the design aspects of creating a slot, while the design considerations for recovery are detailed in that article.

4.2 The Two LSNs You Watch

You read the state of a slot from the pg_replication_slots view. Debezium documentation lists two specific columns to monitor.

The LSN up to which the connector has received data is available in the confirmed_flush_lsn
column of the server's pg_replication_slots view. Data that is older than this LSN is no
longer available, and the database is responsible for reclaiming the disk space.

Also in the pg_replication_slots view, the restart_lsn column contains the LSN of the oldest
WAL that the connector might require.

Here is what each column represents:

Column	Meaning	State to Monitor
`confirmed_flush_lsn`	The position up to which the consumer has acknowledged receipt.	Is it progressing?
`restart_lsn`	The oldest WAL position that the consumer might require.	Is it significantly far from `confirmed_flush_lsn`?

Debezium documentation further explains the situation when these two LSNs are distant from each other:

If the value for confirmed_flush_lsn is regularly increasing and the value of restart_lsn lags
then the database needs to reclaim the space. The database typically reclaims disk space in
batch blocks. This is expected behavior and no action by a user is necessary.

⚠ In other words, a significant difference between the two LSNs is not necessarily an anomaly. The true anomaly is when confirmed_flush_lsn stops progressing. Misinterpreting which metric to monitor can lead to false alerts regarding legitimate recovery delays.

The metrics visible within AWS have different names depending on whether you are using Aurora or RDS. An AWS Database Blog explaining Aurora PostgreSQL parameters states that when a slot begins retaining WAL, the VolumeBytesUsed metric increases, and the TransactionLogsDiskUsage metric can be used to track WAL usage.

4.3 A Slot Is Per Database, but the WAL Is Shared

This is the hardest failure in this article to identify. It arises when a PostgreSQL instance hosts multiple databases, one of which experiences high traffic, while CDC captures a different one that sees much less.

Debezium's official documentation states:

The PostgreSQL instance contains multiple databases and one of them is a high-traffic
database. Debezium captures changes in another database that is low-traffic in comparison to
the other database. Debezium then cannot confirm the LSN as replication slots work
per-database and Debezium is not invoked. As WAL is shared by all databases, the amount used
tends to grow until an event is emitted by the database for which Debezium is capturing
changes.

Taking the structure apart: a slot operates per database, while every database on the instance shares one WAL. If there are no changes occurring in the database being captured, the consumer will not be invoked, and it will be unable to determine its position. During this time, the WAL generated by other databases will continue to accumulate.

In other words, the more inactive the database being captured, the more WAL accumulates. The behavior runs against intuition, and load testing does not reproduce it. It often manifests in production environments, such as when a batch process runs on the uncaptured database during the night.

The official answer is to emit a heartbeat on a schedule. Configure heartbeat.interval.ms to trigger changes in the database being captured. If using a heartbeat table, ensure it is included in a publication.

ALTER PUBLICATION <publicationName> ADD TABLE <heartbeatTableName>;

4.4 The Idle Environment Problem on Amazon RDS

The previous section describes a pattern that has AWS-specific variations. As stated explicitly in the official Debezium documentation:

For users on AWS RDS with PostgreSQL, a situation similar to the high traffic/low traffic
scenario can occur in an idle environment. AWS RDS causes writes to its own system tables to
be invisible to clients on a frequent basis (5 minutes). Again, regularly emitting events
solves the problem.

⚠ It is important to note that the subject of this description is Debezium, not AWS. This article cites this behavior as observed and documented by Debezium. No matching statement was found in the AWS primary documentation. Readers should verify this behavior in their own environments.

The practical implication is clear: This issue only occurs in environments where there is no activity. It is often discovered after setting up a test environment, leaving it idle for a period, and then noticing that storage space is being consumed.

4.5 One Slot, One Consumer

A slot can only occupy one position, and therefore can only track one consumer. Debezium's documentation explicitly outlines the consequences of sharing a slot.

If you permit multiple connectors to capture from a replication slot, you risk data loss,
because a replication slot can emit each change only once.

The documentation also addresses the requirements when multiple connectors are directed toward the same database server.

When you deploy multiple Debezium connectors to capture changes from the same PostgreSQL
database server, each connector requires a unique replication slot and publication.

⇒ When you need to replicate the same changes to two destinations, there are two options: creating two slots, or using a single slot and branching downstream. The former doubles the load on the source side. The latter means the availability of the branching layer becomes dependent on the source's storage. This decision will be discussed in Section 10.

⚠ There is a limit to the number of slots you can create. The max_replication_slots setting sets that limit. The Aurora PostgreSQL user guide states the following regarding how to determine this value:

Set this parameter to a value that's at least equal to your planned total number of logical
replication publications and subscriptions. If you are using AWS DMS, this parameter should
equal at least your planned change data capture tasks from the cluster, plus logical
replication publications and subscriptions.

If you are also running DMS tasks within the same cluster, you must factor in the number of those tasks as well. In environments where separate teams are creating slots independently, this total may not be visible to anyone.

5. Retention Fails in Opposite Directions

5.1 The Same Incident, Two Different Outages

The sections above showed that in PostgreSQL, when consumption stops, the log accumulates. In MySQL, the behavior is the opposite.

One stalled consumer, two opposite outcomes

The RDS user guide states the following regarding binlog retention:

The binlog retention hours parameter is used to specify the number of hours to retain binary
log files. Amazon RDS normally purges a binary log as soon as possible, but the binary log
might still be required for replication with a MySQL database external to RDS.

The default value of binlog retention hours is NULL. For RDS for MySQL, NULL means binary logs
aren't retained (0 hours).

The documentation for Aurora MySQL describes a slightly different behavior.

The default value of binlog retention hours is NULL. For Aurora MySQL, NULL means binary logs
are cleaned up lazily. Aurora MySQL binary logs might remain in the system for a certain
period, which is usually not longer than a day.

This difference in default settings determines the form in which failures manifest.

	Aurora PostgreSQL / RDS for PostgreSQL	Aurora MySQL / RDS for MySQL
When the consumer stops	Logs continue to accumulate.	Logs are discarded.
Symptoms	Source storage exhaustion. Reclamation stops.	Even after the consumer restarts, the necessary logs are missing.
What breaks	The source database.	The CDC path. The database itself is fine.
Recovery method	Drop the slot. This action also cuts the pipeline.	Restore from an initial snapshot.
Items to monitor	The slot's `confirmed_flush_lsn` and storage.	Consumer lag and remaining retention period.

⇒ The same incident — for example, a consumer that stops for three days — can manifest as a failure of the production database in one system, and as data loss in another. It is not possible to definitively say which scenario is less severe. What is clear is that by choosing an engine, you are accepting a particular type of failure.

5.2 Setting the Retention on the MySQL Side

To extend the retention period within a MySQL environment, you need to call a stored procedure.

CALL mysql.rds_set_configuration('binlog retention hours', 24);

The maximum retention period varies depending on the database engine. The following values are specified in the RDS user guide and the Aurora user guide, respectively:

Engine	Maximum Retention Period
RDS for MySQL	168 hours (7 days)
Aurora MySQL 2.11.0 and later, and version 3	2160 hours (90 days)

⚠ Both user guides explicitly state that you cannot specify 0. There is no way to explicitly indicate that you do not want to retain data; instead, data is not retained when the default value of NULL is used.

Both user guides also contain the same warning:

After you set the retention period, monitor storage usage for the DB instance to make sure
that the retained binary logs don't take up too much storage.

⇒ Extending the retention period on the MySQL side introduces the same issues that exist on the PostgreSQL side. The key difference is that in PostgreSQL, data is retained indefinitely until it is consumed, while in MySQL, retention is always cut off after the configured period. Configuring the MySQL side is essentially a process of defining, through a time-based setting, whether you prioritize preventing source shutdown or avoiding data loss.

5.3 What This Means for the Design

Here are three design considerations derived from this asymmetry:

On the PostgreSQL side, the mechanism for detecting consumer failures must be integrated as part of the source availability design. Monitoring failures can directly lead to database downtime, so CDC monitoring cannot be treated as an optional addition.
On the MySQL side, the retention period defines the upper limit for recovery time. If a consumer fails and remains down for longer than the retention period, recovery will require starting from the initial snapshot. The retention period should be understood as a time-based requirement for the recovery process.
In either system, stopping the consumer is not a harmless operation. When deciding to take the consumer down for maintenance, it is necessary to first determine the maximum allowable downtime.

6. Separating Extraction from Delivery

6.1 What the Connector Layer Is For

The connector layer is responsible for transporting changes read from the capture point to their destination. If these two processes are combined, every time a new destination is added, a new capture point is also required. As seen in the previous section, an increase in capture points translates to a greater burden on the source.

The connector layer exists to separate the two. Changes are extracted only once, and the results are written to an intermediate location. Delivery to each destination is then handled by reading from that intermediate location. A prime example of this architecture is the combination of Apache Kafka and Kafka Connect.

AWS offers a managed service called Amazon MSK Connect that provides this architecture. The developer guide states:

MSK Connect is a feature of Amazon MSK that makes it easy for developers to stream data to and
from their Apache Kafka clusters. MSK Connect uses Kafka Connect versions 2.7.1 or 3.7.x,
which are open-source frameworks for connecting Apache Kafka clusters with external systems
such as databases, search indexes, and file systems.

Regarding Change Data Capture (CDC) from databases, AWS specifically mentions Debezium.

You can deploy connectors developed by 3rd parties like Debezium for streaming change logs
from databases into an Apache Kafka cluster, or deploy an existing connector with no code
changes.

⚠ It is crucial to understand the context of this statement. AWS is stating that third-party connectors can be deployed on MSK Connect, and uses Debezium as an example. This does not mean AWS supports Debezium. Debezium is an independent project developed under the Apache License, and its specifications and behavior are primarily documented in Debezium's own documentation. When this article references Debezium's behavior, the source should always be Debezium's documentation.

6.2 The Output Plugin Decides What the Downstream Receives

PostgreSQL logical decoding relies on output plugins to determine the format of the data. The overlap between two separate support lists decides that choice: what the database offers, and what the connector supports.

Connector-Side. The plugins supported by the Debezium PostgreSQL connector are two, according to the official documentation (version 3.6) as of September 5, 2026. The description for the connector property plugin.name states that the default value is decoderbufs.

The name of the PostgreSQL logical decoding plug-in installed on the PostgreSQL server.
Supported values are decoderbufs, and pgoutput.

Database-Side. The plugins listed in the RDS for PostgreSQL user guide are a different set. Furthermore, the number of plugins listed varies between pages. Section 9.2 takes up that discrepancy. For now it is enough to say that there are test_decoding and wal2json, which appear on all pages, and pgoutput, which is listed on only one page.

⇒ Only pgoutput appears on both tables.

Plugin	Database-Side	Debezium-Side	Result
`pgoutput`	Listed on some pages, not on others	Supported	Practical Option
`wal2json`	Listed	Not Supported	For consumers other than Debezium
`test_decoding`	Listed	Not Supported	As the name suggests, for testing
`decoderbufs`	Not Listed	Supported. And the default	Not usable in managed environments

⚠ The last row of this table represents a practical pitfall. The default value for Debezium's plugin.name is decoderbufs, which is not included with PostgreSQL. Because managed database services do not allow the installation of additional plugins, it will not function unless you explicitly set plugin.name to pgoutput. Proceeding with the default value will result in failure.

Regarding pgoutput, the official Debezium documentation states:

The standard logical decoding output plug-in in PostgreSQL 10+. It is maintained by the
PostgreSQL community, and used by PostgreSQL itself for logical replication. This plug-in is
always present so no additional libraries need to be installed.

In managed database services, this statement effectively determines the choice. Requiring no additional libraries is exactly what makes it usable where you cannot install anything.

And the differences in plugins manifest in the downstream code. The official documentation states that the type of values passed to the custom converter can vary depending on the plugin.

decoderbufs passes a byte array (byte[]) representation of the column data.
pgoutput passes a string representation of the column data.

The choice of plugin is not an internal detail of the capture point; it is a premise the downstream conversion code is built on. Changing it mid-process can break the downstream functionality.

⚠ The set of output plugins is not fixed. This article only lists those currently documented in the official documentation as of September 5, 2026. The compatibility tables on both sides are updated independently, so if you are reading this later, you should re-evaluate both the database side and the connector side, recalculating any overlaps yourself. Relying on only one side could lead you to select an incompatible option.

6.3 The Initial Snapshot, and What Restarting Costs

Change streams only capture changes. Data that already exists in the table at the time the process begins will not be captured as changes. Debezium addresses this with an initial snapshot. The official documentation states:

Most PostgreSQL servers are configured to not retain the complete history of the database in
the WAL segments. This means that the PostgreSQL connector would be unable to see the entire
history of the database by reading only the WAL. Consequently, the first time that the
connector starts, it performs an initial consistent snapshot of the database.

This initial snapshot possesses a critically important characteristic for operational purposes. The official documentation explains:

If the connector stops during a snapshot, the connector begins a new snapshot when it restarts.

⚠ Do not resume from an interrupted state; start from the beginning. In environments with large tables, this presents a practical limitation. If a table requires several hours to capture the initial snapshot, and restarts occur repeatedly, the system may never reach a stable state.

Once the initial snapshot is complete, the connector records this in its offset, which will be the subject of the next section.

6.4 Where the Resume Position Lives

When a connector restarts, it needs to know where to resume reading. This position is not stored within the connector's process itself. The MSK Connect developer guide explicitly states where this state is located.

By default, Amazon MSK Connect creates three separate topics in the Kafka cluster for each
Amazon MSK Connector to store the connector's configuration, offset, and status.

⇒ The state is stored in a Kafka topic. This is what makes it fair to call the connector layer a separation of extraction from delivery. The executing process can be disposable, leaving only the position. The MSK Connect developer guide states the following regarding tasks:

Tasks don't store state, and can therefore be started, stopped, or restarted at any time in
order to provide a resilient and scalable data pipeline.

The default number of partitions for internal topics varies depending on the use case.

Internal Topic	Default Number of Partitions	Modifiable
`config.storage.topic`	1	Not modifiable. Must be a single partition.
`offset.storage.topic`	25	Can be modified using `offset.storage.partitions`.
`status.storage.topic`	5	Can be modified using `status.storage.partitions`.

Debezium clearly outlines the behavior of Kafka Connect in the event of a failure. The outcome differs depending on whether the process shuts down gracefully or crashes.

If the Kafka Connector process stops unexpectedly, any connector tasks it was running
terminate without recording their most recently processed offsets. When Kafka Connect is being
run in distributed mode, Kafka Connect restarts those connector tasks on other processes.
However, PostgreSQL connectors resume from the last offset that was recorded by the earlier
processes. This means that the new replacement tasks might generate some of the same change
events that were processed just prior to the crash. The number of duplicate events depends on
the offset flush period and the volume of data changes just before the crash.

⇒ The offset flush interval decides how many duplicates you can get. This is a configurable value, representing a trade-off chosen by the designer. Writing offsets more frequently reduces the potential for duplication, but increases the overhead associated with writing those offsets.

Furthermore, connector and task restart functionality was added to MSK Connect on August 31, 2026. This allows you to select and restart only the failed tasks.

6.5 The Connector Name Is Part of the Contract

The sections above established that the position lives in a Kafka topic. So, how are the topic and connector linked? The MSK Connect developer guide states something critically important in practice.

If you want to reuse the offset storage topic to consume offsets from a previously created
connector, you must give the new connector the same name as the old connector.

⇒ The connector name is the key to inheriting the position. When you recreate a connector and change its name, you will not be able to inherit the position. Failing to inherit the position means you will have to start from the initial snapshot.

⚠ This will not manifest as a configuration error. The new connector will start and run correctly. However, it will effectively restart from the beginning. With large tables, it can be difficult to notice this.

The same page also states that configuration and status topics cannot be shared.

MSK Connect does not allow different connectors to share config.storage.topic and
status.storage.topic. Those topics are created each time you create a new connector in MSKC.

Furthermore, it notes that internal topics associated with a deleted connector are not automatically removed.

Old topics that are attached to deleted connectors are not automatically removed because
internal topics, such as offset.storage.topic, can be reused among connectors.

⇒ Every time you recreate a connector, the number of unused topics will increase. Cleaning up these topics is the responsibility of the operations team. The format of the topic names used during cleanup is inconsistent within the original documentation, a detail Section 9 takes up.

7. Two Capture Points on the Same Table

7.1 The Choice That Looks Like a Retention Question

There are two ways to extract changes from a DynamoDB table: DynamoDB Streams and Kinesis Data Streams for DynamoDB.

Comparisons between these two options often focus on retention periods and their respective ecosystems. DynamoDB Streams offers a 24-hour retention period and can be read from Lambda and the Kinesis Client Library. Kinesis Data Streams offers a retention period of up to 365 days and integrates with services like Amazon Data Firehose and Amazon Managed Service for Apache Flink. While this comparison is accurate, it does not highlight the crucial differences.

7.2 The Guarantees Are Not the Same

The DynamoDB developer guide states the following regarding DynamoDB Streams:

DynamoDB Streams helps make sure the following:

Each stream record appears exactly once in the stream.
For each item that is modified in a DynamoDB table, the stream records appear in the same
sequence as the actual modifications to the item.

The same developer guide, on the page for Kinesis Data Streams for DynamoDB, states:

The Kinesis data stream records might appear in a different order than when the item changes
occurred. The same item notifications might also appear more than once in the stream. You can
check the ApproximateCreationDateTime attribute to identify the order that the item
modifications occurred in, and to identify duplicate records.

⇒ For changes to the same table, one guarantees no duplication and per-item order, and the other guarantees neither.

	DynamoDB Streams	Kinesis Data Streams for DynamoDB
Duplication	Records appear only once within the stream.	The same notification may appear multiple times.
Order	Per item, the same order as the actual changes.	May appear in an order different from when the changes occurred.
Method for Restoring Order	Not required.	Examine the `ApproximateCreationDateTime` attribute.
Retention Period	24 hours	Default: 24 hours. Maximum: 8760 hours (365 days).
Concurrent Readers	Up to 2 processes per shard. Subject to conditions.	Supports two methods: shared fan-out and enhanced fan-out.

⚠ The number of readers is subject to a condition that is easily overlooked. The DynamoDB developer guide states:

For single-Region tables that are not global tables, you can design for up to two processes to
read from the same DynamoDB Streams shard at the same time. Exceeding this limit can result in
request throttling. For global tables, we recommend you limit the number of simultaneous
readers to one to avoid request throttling.

⇒ The value of 2 is only applicable to tables within a single Region, and not global tables. For global tables, 1 is recommended. An eventually consistent global table is built on replication that reads the stream itself, which is why less of that capacity is left for consumers. Section 7.3 covers this. You may observe this by seeing a second consumer that was previously working begin to be throttled after migrating to a multi-region configuration.

⚠ The scope of the guarantees is also subject to conditions. The order guarantees provided by DynamoDB Streams apply at the item level, not at the partition level. The developer guide clarifies this as follows:

DynamoDB Streams guarantees ordering at the level of an individual item—that is, across all
modifications to the same primary key (the partition key, or the partition key and sort
key)—not across an entire partition. Because an item collection that shares a partition key
can span more than one partition, DynamoDB preserves the order of changes for each item rather
than across a whole item collection.

⇒ There is no guarantee of order between multiple items that share the same partition key. The guarantee only applies to the order of changes for a single item that shares the same primary key.

7.3 On a Global Table, the Guarantee Changes Again

The guarantees discussed in the previous sections relate to tables within a single Region. In a global table, the same DynamoDB Streams behave differently. Furthermore, this behavior varies depending on the consistency mode.

The DynamoDB developer guide states the following regarding global tables with eventual consistency:

Global tables configured for multi-Region eventual consistency (MREC) replicate changes by
reading those changes from a DynamoDB Stream on a replica table and applying that change to
all other replica tables. Streams are therefore enabled by default on all replicas in an MREC
global table, and cannot be disabled on those replicas.

⇒ In this configuration, the stream is not a feature for users, but rather an implementation of replication itself. It is not even something you choose to turn on or off, and it cannot be turned off. The limitations on readers discussed in Section 7.2 were restricted to tables within a single Region, precisely because of this.

This configuration's stream also exhibits behavior that influences downstream design.

The MREC replication process might combine multiple changes in a short period of time into a
single replicated write, resulting in each replica's Stream containing slightly different
records. Streams records on MREC replicas are always ordered on a per-item basis, but ordering
between items might differ between replicas.

⚠ Here, two distinct points are made.

Multiple changes occurring within a short period may be aggregated into one. This means intermediate states may not appear in the stream. For example, if a value changes from A to B and then to C, the downstream system might only see C. Downstream systems that require all transitions may silently miss data.
The contents of the stream differ between replicas. When reading the same table and the same change in different Regions, you may obtain different records. The streams in the two regions are not compatible.

In global tables with strong consistency, this is reversed.

Global tables configured for multi-Region strong consistency (MRSC) do not use DynamoDB
Streams for replication, so Streams are not enabled by default on MRSC replicas. You can
enable Streams on an MRSC replica. Streams records on MRSC replicas are identical for every
replica, including Stream record ordering.

⇒ To summarize:

	Single-Region Table	MREC Global Table	MRSC Global Table
Stream Role	A feature that users can enable	An implementation of replication; cannot be disabled.	A feature that users can enable. Disabled by default.
Change Aggregation	Not specified	Multiple changes within a short period may be aggregated.	Not specified
Replica Consistency	Not applicable	The contents may differ slightly between replicas.	Identical across all replicas, including order.

⚠ The significance of this table is that the guarantees at the output change in three stages. It is not just about which stream you choose (Section 7.2), but also about how the table is configured. Issues may arise when downstream systems, initially designed and validated for a single region, begin to miss data after migrating to a global table.

7.4 How to Choose

Considering these differences, the criteria for selection should no longer be based solely on retention period.

Scenario	Selection	Reason
Downstream components rely on item-level ordering	DynamoDB Streams	Order restoration never has to enter the downstream at all.
Downstream components cannot handle duplicates, or processing duplicates is costly	DynamoDB Streams	Eliminates the need to implement duplicate elimination.
A recovery time exceeding 24 hours is required	Kinesis Data Streams	Allows for extended retention periods.
Three or more consumers need to read the same changes	Kinesis Data Streams	Different limits on the number of readers per shard.
Connecting to Firehose or Flink	Kinesis Data Streams	Connections are readily available.

⚠ The top and bottom halves of the table above are often required simultaneously. Requiring both ordering and a retention period beyond 24 hours is a common combination. In such cases, Kinesis Data Streams should be selected, and ordering restoration and duplicate elimination should be implemented in the downstream components. This involves trading off increased complexity in the downstream system to achieve the desired retention period. It is valuable to explicitly acknowledge this trade-off during the design phase.

7.5 One More Thing the Stream Does Not Show

DynamoDB Streams have another behavior that impacts downstream design. The developer guide states:

If you perform a PutItem or UpdateItem operation that does not change any data in an item,
DynamoDB Streams does not write a stream record for that operation.

⇒ Writes that do not change a value will not appear in the stream. For downstream applications that want to count the number of writes, this will result in a gap. The change stream only reports changes in values, not the act of writing itself. This characteristic is inherent to the stream's nature, and there is no way to recover this information downstream.

8. Reading the Guarantee at the Capture Boundary

8.1 Three Segments, and Which One This Article Owns

The greatest challenge when reading articles about delivery guarantees is the lack of clarity regarding which segment they are discussing. The process is divided into at least three segments.

Segment	What Happens	Who Handles It
Segment 1: Extraction	Changes to the database enter the process via the capture point.	This article
Segment 2: Delivery	The data is transported through the process, passing through intermediate locations before reaching its destination.	Covered by existing articles.
Segment 3: Downstream Application	The destination receives the data and applies it to its own state.	Covered by existing articles.

This article focuses solely on Segment 1. Segments 2 and 3 — specifically, the delivery guarantees and the design for idempotency and duplicate elimination required downstream — are covered in three documents mentioned in Section 1.2. The breakdown of order, duplicates, and exactly-once processing is discussed in Section 7 of AWS Real-Time Streaming Data Pipeline Architecture Guide - Ingestion, Processing, and Delivery with Kinesis, Managed Service for Apache Flink, and OpenSearch. Section 9 of AWS Messaging and Event Routing Decision Guide - Choosing Between SQS, SNS, EventBridge, and Kinesis addresses consumer-side design based on an at-least-once assumption, and Event-Driven Architecture Anti-Patterns on AWS - Failure Modes, Root Causes, and How to Design Around Them covers failure modes when order is assumed.

⚠ Avoid discussing guarantees without specifying the segment. Statements about exactly-once delivery cannot be verified without clearly identifying the relevant segment. The absence of duplicates in Segment 1 and the prevention of double application in Segment 3 are separate claims.

8.2 What Each Capture Point Promises at the Boundary

When limited to Segment 1, the three capture points provide the following guarantees:

Capture Point	Guarantee at the Boundary	What the Source States
DynamoDB Streams	No duplicates. Item-level order.	Appears only once within the stream, and changes to the same item appear in actual order.
Kinesis Data Streams for DynamoDB	Guarantees neither of those two.	Order may differ. The same notification may appear multiple times.
WAL via Logical Decoding	Delivered only once up to the slot position.	Each slot can output a change only once.
binlog	Position-based reads.	Readable only within the range it retains.

⚠ Please note the meaning of the fourth row. The binlog provides a description of availability, not a guarantee. The retention setting decides what is still readable, and that is a different thing from a guarantee. As seen in Section 5, the design of MySQL is based on time, not guarantees.

8.3 What the Capture Boundary Hands to the Next Segment

Segment 1 passes not only change records to Segment 2, but also requirements outlining what Segment 2 must do.

Capture Point	Requirements Passed to Segment 2 and Beyond
DynamoDB Streams	No additional processing is required regarding order and duplicates.
Kinesis Data Streams for DynamoDB	Order restoration using `ApproximateCreationDateTime` and duplicate detection are necessary.
Logical Decoding via Debezium	Resilience against duplicates during crash recovery is required.

Regarding the final row, Debezium explicitly states:

Because there is a chance that some events might be duplicated during a recovery from failure,
consumers should always anticipate some duplicate events. Debezium changes are idempotent, so
a sequence of events always results in the same state.

This statement conveys two points. Consumers should anticipate duplicates, and Debezium's change events, when applied in the same order, will result in the same state. However, this does not render duplicates harmless. While applying the same event twice will result in the same state if the downstream system applies the value directly, duplicates will alter the result if the downstream system applies an increment.

⇒ This is where the choice of capture point impacts the downstream design. How to design for it in concrete terms is covered by the published articles named in Section 1.2.

9. Where the Primary Sources Disagree

9.1 Why This Section Exists

Primary sources are split across pages, and they sometimes say the same thing in different words. This section documents instances where choosing one option over another, when a user is actually writing a command, can lead to different results.

9.2 Two Pages Disagree on Which Plugins Are Supported

Regarding the output plugins discussed in Section 6.2, two pages in the RDS for PostgreSQL user guide list different sets of supported plugins.

The page describing the core functionality of logical replication states:

Currently, RDS for PostgreSQL supports the test_decoding and wal2json output plugins that ship with PostgreSQL.

The page describing logical replication in a Multi-AZ DB cluster states:

Currently, RDS for PostgreSQL supports the test_decoding, wal2json, and pgoutput plugins that ship with PostgreSQL.

Both pages use the same subject, but one lists two plugins while the other lists three. The difference is pgoutput.

⚠ Furthermore, the core functionality page's own example contradicts its listing. Later in the same page, a demonstration uses CREATE PUBLICATION to configure native logical replication, and the resulting output displays the contents of pg_replication_slots. The plugin column in that output shows pgoutput. A plugin not listed in the initial listing appears in the results presented on the same page.

⇒ The Multi-AZ page appears to be more accurate, and the listing on the core functionality page is incomplete. This article bases its judgment on the example provided on the core functionality page itself.

⚠ This discrepancy has practical consequences. A designer who only reads the core functionality page might conclude that pgoutput is not supported and instead choose wal2json. However, as Section 6.2 showed, Debezium does not support wal2json. Consequently, following the information in the primary source leads to an incompatible configuration.

9.3 Two Names for the MSK Connect Internal Topics

Regarding the names of the internal topics discussed in Section 6.4, two pages in the MSK Connect developer guide use different formats.

The page describing state management reads:

__msk_connect_configs_connector-name_connector-id
__msk_connect_status_connector-name_connector-id
__msk_connect_offsets_connector-name_connector-id

The page describing offset management reads:

The internal topics are named following the format
__amazon_msk_connect_<offsets|status|configs>_connector_name_connector_id.

The prefixes differ. One uses __msk_connect_, while the other uses __amazon_msk_connect_.

A third page, which provides concrete examples for describing the default offset topics, uses the latter format.

__amazon_msk_connect_offsets_my-mskc-connector_12345678-09e7-4abc-8be8-c657f7e4ff32-2

⇒ __amazon_msk_connect_ appears to be the dominant form, as demonstrated in the examples. However, this article does not make a definitive statement. Please verify this by retrieving the topic list in your actual environment.

⚠ This discrepancy can cause issues. One page provides a regular expression for deleting obsolete internal topics. If the prefix is incorrect, this regular expression will not match any topics, or it may match unintended topics. Because this operation involves deletion, always list and visually confirm the matching topic names before executing it.

9.4 What Is Documented by Whom

Another point to record is the difference in the nature of the sources. The growth of the WAL in an idle Amazon RDS environment, cited in Section 4.4, is stated in Debezium's documentation, and was not found in the AWS documentation.

This is not a contradiction, but it is a distinction that readers should be aware of. Behavior not documented in AWS's documentation may change without notice due to modifications on AWS's side. It is best to treat this item as something to verify in your own environment and to periodically re-verify.

10. What This Path Takes Away

Deciding to run your own CDC layer buys you something, and it costs you something. Without this section, this article would simply become promotional material.

10.1 The Source Database Stops Being Only Yours

As soon as you open a capture point, external factors begin to impact the source database. As Sections 4 and 5 showed, the state of the consumer can affect the storage of the source data. This can occur even with a zero-ETL approach, but the key difference with a layer you build is that you are responsible for designing the availability of the consumers.

Specifically, the option of taking consumers offline to observe their behavior is lost. On the PostgreSQL side, the source might run out of storage while the consumers are offline. On the MySQL side, necessary logs might be deleted while the consumers are offline. In either case, it is necessary to set a time limit before taking the consumers offline.

10.2 Some Operations on the Source Become Coordinated Operations

Certain operations on the source system now require coordination with the CDC path.

Major Version Upgrades: It is necessary to predetermine how slots will be handled.
Parameter Changes: Changes to parameters that enable logical replication will require a restart.
Schema Changes: The appearance of DDL statements may vary depending on the capture point. It is necessary to determine how downstream systems will handle changes to columns.

10.3 The Initial Snapshot Becomes a Recurring Cost

As seen in Sections 6.3 and 6.5, the initial snapshot is not a one-time operation. It happens again when you rebuild a connector, when you change its name, and when a snapshot stops partway. For large tables, this repeated retrieval can become a significant operational limitation.

10.4 You Now Own the Ordering Question

In managed integrations, the responsibility for handling order and duplicates lies with the provider. However, when using your own layer, this becomes a matter of your own design choices. As Section 7 showed, the choice of capture point has already decided part of this process. When you prioritize retention and choose Kinesis Data Streams, restoring order becomes a downstream task.

10.5 When to Stay With the Managed Path

Given all of the above, it is worth going back to the decision not to run this layer yourself when any of the following holds.

The destination is listed in the zero-ETL mapping table, and transformations can be executed downstream.
The system for detecting and addressing consumer failures does not meet the availability requirements of the source database.
Retaking the initial snapshot takes longer than the acceptable timeframe for business operations.

⚠ The third point is often overlooked. This issue can manifest as a situation where a system designed for smaller tables becomes impractical to reacquire due to the table's size after several years.

11. Failure Modes

This section lists failure modes that do not appear during mechanical inspections but emerge after a period of operation. The descriptions will be presented from the perspective of the symptoms.

11.1 Storage Draining on an Idle Verification Environment

You build a verification environment, run changes through the CDC path, confirm it works, and leave it. Days later, storage on the source has shrunk.

Cause: This behavior is related to the scenarios described in Sections 4.3 and 4.4. Because there are no changes being captured, the slot's position does not advance. During this time, other databases and system tables continue to consume WAL (Write-Ahead Logging) space. Applying a load makes it go away, because the absence of changes is what causes it. This behavior can delay the identification of the root cause.

What to check: Whether the confirmed_flush_lsn value in pg_replication_slots is advancing. If it is not advancing, verify that there are truly no writes occurring to the capture target.

11.2 Rebuilding a Connector and Silently Re-Snapshotting

Review and recreate the connector settings. The new connector starts up successfully and produces no errors. However, a large volume of outdated data is being sent to the destination.

Cause: This is due to the type described in Section 6.5. Because the connector's name changed, it cannot pick up the position from the offset storage topic, so it starts again from the initial snapshot. This is not an error, so it is not logged.

What to check: Before recreating the connector, record the name of the old connector. If you need to change the name, explicitly specify offset.storage.topic to ensure it refers to the same topic.

11.3 Slow Consumer Producing Opposite Symptoms on Two Engines

The same setup is built on both PostgreSQL and MySQL, and in both cases the consumer falls behind. On the PostgreSQL side a storage alarm fires, and on the MySQL side nothing does. You conclude that the MySQL side is fine.

Cause: This is related to the type defined in Section 5. On the MySQL side, logs are already being discarded. Nothing fires because nothing is accumulating, not because everything is healthy. When the consumer resumes, it is discovered that the necessary data is no longer available.

What to check: On the MySQL side, monitor the consumer slowdown itself, rather than storage capacity. It is important to clearly document, from the design stage, that the monitoring targets differ depending on the engine.

11.4 Counting Writes That Produce No Change

DynamoDB Streams is used to count the number of writes. The count obtained is lower than the number recorded on the application side.

Cause: This is due to the type described in Section 7.5. Writes that do not result in a change are not reflected in the stream. There is no way to recover this data downstream. If you need to count the total number of writes, you will need to use a method other than the change stream.

11.5 Adding a Second Consumer to an Existing Slot

You want the same change delivered to a second destination, so you point another connector at the existing slot. Both look like they are running, and both destinations end up missing rows.

Cause: This is due to the pattern described in Section 4.5. The slot only outputs each change once. This results in both consumers sharing the same data. Because both are active, the system does not detect that one consumer has stopped receiving data.

What to check: When adding consumers, first decide whether to increase the number of slots or to branch the output from a single slot downstream.

11.6 Losing Intermediate States After a Move to a Global Table

The system combines change streams from a single region's table and tracks value transitions downstream. This works without issues in both validation and production environments. However, when the table is migrated to a global table, the downstream components begin to miss transitions.

Cause: This is the type described in Section 7.3. In an eventually consistent global table, short bursts of multiple changes may be consolidated into a single replicated write. These consolidated intermediate states are not reflected in the stream. This is a missing element, not an error, so no alerts are triggered.

What to check: Determine, during the design phase, whether the downstream components require all transitions or only the final state. If the former is the case, migrating to a global table will conflict with this requirement.

11.7 Reading a Guarantee Without Its Segment

In a design review, the path is described as exactly-once, and downstream deduplication is left out. Duplication occurs in the production environment.

Cause: This is the type described in Section 8.1. The segment was never checked. The fact that there is no duplication within a specific segment does not guarantee that there is no duplication across the entire path.

What to check: When reviewing any guarantee, always check the source and the segment it applies to. If the source does not name a segment, that statement cannot be used as justification for the design.

12. Frequently Asked Questions

Does a self-managed CDC layer become mandatory if zero-ETL integration does not support a particular source and target combination?

No. First, consider using AWS DMS. If DMS supports the source and target pair, and DMS transformation rules can express the transformations you need, it is lighter to operate than a layer you build. A layer you build should be considered as an option after evaluating DMS.

Does the source stay unaffected as long as I never create a replication slot?

No. Once rds.logical_replication is enabled, additional WAL (Write-Ahead Log) files will begin to be written before a slot is actually created. The RDS for PostgreSQL user guide notes that enabling this parameter can increase WAL generation, and recommends only setting it when using logical slots. Not creating a slot avoids the need to retain WAL files until they are consumed.

Is there a way to prevent the source storage from continuing to grow while the consumer is stopped?

You can stop it by deleting the slot. However, deleting the slot will result in the consumer's position being lost, so when you resume the pipeline, you will need to start from an initial snapshot. The decision to stop or not should be based on a comparison of the cost of stopping and re-capturing the source.

Why is the WAL increasing even though the database being captured is idle?

Slots operate at the database level, but WAL files are shared across all databases within the instance. Even if there are no changes to the captured database, the consumer cannot confirm its position, and WAL files from other databases may accumulate during that time. Debezium recommends periodically sending heartbeat signals to address this situation.

Is Debezium an officially supported product by AWS?

No. Debezium is an independent, open-source project. While the AWS developer guide mentions Debezium as an example of a third-party connector that can be deployed within MSK Connect, this simply indicates that it can be deployed, and does not mean that AWS guarantees its specifications or behavior. The primary source of information regarding its behavior is Debezium's own documentation.

Can I change the output plugin later?

Changing it can break the downstream. Debezium's documentation states that decoderbufs passes column values as byte arrays, while pgoutput passes them as string representations. This is because downstream transformation code often relies on these specific data types.

What should I be careful about when rebuilding the connector?

It is about names. The MSK Connect developer guide states that to inherit the offset from a previous connector, you need to give the new connector the same name. Change the name and it cannot inherit the position, so it starts again from the initial snapshot. Because no errors are generated, it can be difficult to notice this issue.

Are DynamoDB Streams and Kinesis Data Streams for DynamoDB different besides their retention periods?

Yes. They differ in their guarantees regarding order and duplication. The DynamoDB developer guide states that with DynamoDB Streams, records appear only once within the stream and maintain item-level order. However, for Kinesis Data Streams, it notes that order may not be guaranteed and that the same notification could appear multiple times. Choosing based solely on retention period might lead you to later discover that you need to implement order restoration and deduplication downstream.

Does DynamoDB Streams' order guarantee apply even between items with the same partition key?

No. The developer guide clarifies that the guarantee only applies to the order of changes for a single item with the same primary key, not an entire collection of items. This is because a collection of items sharing the same partition key might span multiple partitions.

If I extend the binlog retention period on the MySQL side, will it be safer than on the PostgreSQL side?

It only changes the form of failure. Extending the retention period will also cause the MySQL side to consume more storage. Both the RDS and Aurora user guides recommend monitoring storage usage after setting a retention period. The difference is that with PostgreSQL, data is retained indefinitely until it is consumed, while with MySQL, it is always cut off after the configured period.

Is this pipeline exactly-once?

That cannot be answered without naming a segment. Ensuring that there are no duplicates within the extraction segment and ensuring that downstream components do not apply operations twice are separate claims. This article only addresses the extraction segment. For design considerations downstream of delivery, see the published articles named in Section 1.2.

When pointing Debezium to RDS or Aurora PostgreSQL, is it okay to leave plugin.name at its default value?

No. Debezium's default plugin.name is decoderbufs, which is not included with PostgreSQL and cannot be installed as an additional plugin in managed database services. You must explicitly set it to pgoutput.

Can wal2json be used with RDS for PostgreSQL?

The RDS for PostgreSQL user guide lists wal2json as a supported plugin. However, the Debezium PostgreSQL connector does not support wal2json. What the database offers and what your connector supports are two different things. Compare both lists. Sections 6.2 and 9.2 carry the detail.

Does using global tables affect the guarantees provided by DynamoDB Streams?

Yes, it does. The developer guide states that with eventually consistent global tables, multiple changes within a short period may be combined into a single replication write, and the content of the stream may vary slightly between replicas. For strongly consistent global tables, the guide states that the stream records are identical across all replicas, including their order. Assumptions validated with single-Region tables may not always hold true.

Do write operations that do not change a value appear in the change stream?

No, they do not appear in DynamoDB Streams. The developer guide states that DynamoDB Streams does not generate stream records for PutItem or UpdateItem operations that do not modify the item's data. The change stream reflects changes in values, not the act of writing itself.

13. Summary

This article addressed the handling of database change streams when neither a traditional ETL process nor a migration tool is in use. Key points include:

The choice is not the brand of tool; it is where you put the capture point. Based on the records written by the database itself, the patterns fall into three categories: reading via logical decoding of the WAL, reading binlog, or reading a native stream. The large number of tools appears because they all ultimately read from one of these three sources.
A replication slot is an operational object, not a setting. The existence of a slot means you are instructing the database not to discard WAL files that have not yet been read. The key columns to monitor are confirmed_flush_lsn and restart_lsn in the pg_replication_slots view.
A slot works per database, while the WAL is shared across the instance. Counterintuitively, the quieter the capture process, the more WAL files accumulate. Debezium notes that this can occur even in seemingly quiet environments on Amazon RDS.
Retention fails in opposite directions depending on the engine. In PostgreSQL, if consumption stops, logs accumulate, potentially exhausting the source. In MySQL, by default, logs are not retained, so if consumption lags, the necessary data may be lost. The same underlying issue can result in a production database outage in one case and data loss in another.
The connector layer separates extraction from delivery. The position does not live in the process; it lives in a Kafka topic. And in MSK Connect, the key to that handover is the connector's name. Recreating it under a different name makes it start over from the initial snapshot, and it does so without raising an error.
The two capture points on the same DynamoDB table give opposite guarantees. DynamoDB Streams guarantees uniqueness and item-level ordering, while Kinesis Data Streams for DynamoDB provides neither. Choosing based solely on retention period and ecosystem can lead to increased complexity downstream.
That guarantee shifts once more with how the table itself is configured. In an eventually consistent global table, changes may be batched, potentially losing intermediate states, and the content of the stream may differ between replicas. Assumptions validated in a single-Region environment may not hold true when applied more broadly.
There are two support lists, and only the overlap is usable. One is the set of output plugins the database offers, the other is the set the connector supports. Furthermore, these lists sometimes contradict each other in official documentation. Relying on only one list can lead to incompatible combinations.
Read a guarantee against a named segment. This article covers only the extraction segment. Delivery and downstream application are covered by the existing articles. Any description of guarantees that does not name its segment cannot be used as a basis for design.

A decision to run this layer starts from what the destination wants, and lands on how the source is operated. Ultimately, the final determination is not about what is delivered to the recipient, but rather about what the source database contains.