Who Assigns Shards and Partitions to a Stream Consumer on AWS - KCL Leases, Kafka Group Protocols on Amazon MSK, and Lambda Event Pollers

First Published: 2026-11-02
Last Updated: 2026-11-02

When multiple consumers are reading from a stream, some entity determines which consumer reads which shard or partition. When using the Kinesis Client Library (KCL) to read Kinesis Data Streams, the workers that the user runs make this determination, and the results are stored in a DynamoDB table. In this article's reading, that table is in the account that runs the KCL. When reading Apache Kafka, the group coordinator in the brokers manages the group, and the location where assignments are calculated differs between the classic protocol (the client that becomes the group leader) and the newer consumer protocol (the broker). With AWS Lambda's event source mapping (ESM), AWS-side event pollers handle the reading, and the sources barely describe how assignments are determined or where that state is stored.

This article presents these three methods—KCL, Kafka, and Lambda ESM—in a table, with columns detailing who makes the decision, where the state is stored, what triggers a reassignment, and what the consumer observes during that process. This article also shows what happens when you increase the number of consumers, when one consumer fails, when a new version is deployed, or when consumers share a consumer group ID. It shows what stops and where duplicates or delays appear, in the same columns. This article separates what the sources state from what was not found in them. Conclusions drawn from combining information across multiple sources are presented as this article's reading.

This article is neither a step-by-step guide for using KCL, Kafka clients, or Lambda, nor is it a performance comparison. It will not recommend which consumer to choose, nor will it discuss pricing. The information presented here was verified as of November 1, 2026.

Related articles on this site:

1. Scope and Assumptions
2. How to Read the Tables
3. Three Places That Decide the Assignment
4. KCL — Assignment Recorded in Your Lease Table
5. Kafka — Rebalancing Managed by the Group Coordinator
6. Lambda Event Source Mappings — Read by AWS-Side Event Pollers
7. What a Consumer Sees When the Group Changes
8. What This Setup Showed
9. Where the Sources Disagree and What They Do Not State
10. Failure Modes
11. Frequently Asked Questions About Stream Consumer Assignment
12. Summary
13. References

1. Scope and Assumptions

This section will first define the intended audience, the scope of this article, the source materials consulted, and how to distinguish between different entities that may share the same name. The conclusions, presented in advance, are as follows:

In KCL, the workers that the user runs decide which worker holds which shard. The results of this decision are recorded in a row within a DynamoDB table called the lease table. KCL 3.x creates three metadata tables by default, but from version 3.5, these can be consolidated into one table with the single table format. While a single sentence describing the lease table residing in the user's account could not be found, this article derives that understanding from three separate statements (Section 4.2).
The default assignment strategy in KCL 3.x aims to equalize CPU utilization across workers, rather than to equalize the number of leases. When CPU utilization cannot be collected, assignments are made based on the throughput of each worker. While the implementation and CHANGELOG indicate that from version 3.4.3, a strategy based on the number of leases can be selected through configuration, the developer guide's configuration table does not list this setting (Section 4.4).
The graceful lease handoff feature in KCL 3.x reduces, but does not eliminate, the possibility of a record being processed twice during lease transitions. The developer guide recommends building downstream components to be idempotent.
In Kafka, a group coordinator within a broker manages a collection of consumers. In the classic protocol, the client that becomes the group leader calculates the assignments. In the consumer protocol defined in KIP-848, the broker calculates the assignments, and only the affected partitions move.
From Kafka 4.0 onwards, the consumer protocol is enabled by default on the server side, but the default for the client-side group.protocol remains classic. The Large Groups Blog states that KIP-848 became the default, while the Next-Generation Protocol Blog and the Kafka 4.2 documentation state that it is not enabled by default. This article reads this disagreement as a difference in whether the posts refer to the server-side or client-side default.
In Amazon MSK, the consumer protocol is available on both Standard and Express brokers across all versions 4.x, according to the Next-Generation Protocol Blog. In this article's reading, this means the 4.x versions that each broker type offers: the MSK versions page lists Kafka 4.2 as available only on Express brokers (Section 5.6). No statement was found on a configuration that uses share groups (KIP-932) in production on MSK (Section 5.9).
Regarding Lambda's Kafka ESM and Stream mode, the Lambda developer guide states, within a description of two modes for self-managed Kafka, that Lambda assigns each partition to a single consumer. No statement was found on whether this assignment is determined by the Kafka coordinator or within Lambda itself (Section 6.2).
No statement was found on where Lambda's Kinesis and DynamoDB Streams ESMs store the state indicating how far each shard has been read (Section 6.7). About this state, the sources only mention that the ESM is managed as a resource within the Lambda service.

1.1 Intended Audience

This article is intended for those responsible for applications and platforms that use Kinesis Data Streams, Amazon MSK, and DynamoDB Streams, and which read data using either the KCL, Kafka clients, or Lambda functions. Specifically, this article targets readers who believe one of the following statements to be true:

Increasing the number of consumers will result in shards being distributed evenly across those consumers.
Even if a consumer fails, another consumer will quickly take over, ensuring that the same record is never processed twice.
Lambda divides shards with the same mechanism as KCL.
Upgrading to Kafka 4 will introduce a new rebalancing protocol.

The validity of each of these assumptions can vary depending on the version, configuration, broker type, and the consumer layer involved.

1.2 What This Article Covers and Does Not Cover

This article focuses on the decisions involved in determining the owner of shards and partitions. Specifically, it covers the following:

Metadata tables, lease assignments, failover, graceful lease handoff, migration from KCL 2.x to 3.x, the phase of KCL 1.x, and versions of the DynamoDB Streams Kinesis Adapter.
Kafka's group coordinators, including both the classic and consumer protocols, server-side and client-side defaults, differences in broker configurations between MSK Standard and Express, static membership, the limit for large groups, and share group functionality.
Information regarding what is and is not documented concerning assignment within Lambda's Kafka, Kinesis, and DynamoDB Streams ESM.
Ways to check the assignment yourself.

This article does not cover the following.

Pricing and billing mechanisms.
Where a stream consumer can start on AWS. This topic is covered in Where a Stream Consumer Can Start on AWS. This article does not cover it further.
Configuring the number of Lambda provisioned mode pollers. This topic is covered in Lambda Event Pollers in Provisioned Mode for Kafka and SQS.
MSK broker types and storage tiers, and Intelligent Rebalancing for Express brokers. This topic is covered in Amazon MSK Broker Types and Storage Tiers.
Ordering guarantees, service by service on AWS. This topic is covered in What Breaks Ordering, Service by Service on AWS.
What happens to a failed batch in a Lambda event source mapping. The related article What Happens to a Failed Batch in a Lambda Event Source Mapping covers this topic.
Where a Kinesis consumer resumes after a shard splits or merges. The related article Where a Kinesis Consumer Resumes After a Shard Splits or Merges covers this topic. This article will only address the fact that leases are created for child shards during splits or merges.
Kafka authentication, registration limits for enhanced fan-out, shard assignment for Managed Service for Apache Flink, and consumer operations on Amazon EKS.

1.3 Date of Review and Materials Reviewed

This article was written after reviewing the following materials on November 1, 2026. Abbreviations enclosed in parentheses will be used throughout the text.

Amazon Kinesis Data Streams Developer Guide, specifically the chapters on the KCL (Kinesis Client Library), including: the concepts page (the KCL concepts page), the metadata tables and assignment page (the KCL tables page), the configuration page (the KCL configuration page), the migration page from 2.x to 3.x (the KCL migration page), the rollback page, the single table format page, the IAM permissions page, the version lifecycle page (the KCL lifecycle page), the information page for KCL 1.x and 2.x (the KCL 1.x and 2.x page), and the document history.
Amazon DynamoDB Developer Guide, specifically the pages on DynamoDB Streams and the Kinesis Adapter (the Adapter page), and the migration page from KCL 1.x to 3.5.x+ (the DynamoDB KCL migration page).
Amazon MSK Developer Guide, specifically the pages on supported Kafka versions (the MSK versions page), troubleshooting, version support, and the read-only configuration for Express brokers (the Express configuration page).
AWS Lambda Developer Guide, specifically the pages on Kafka consumer group ID (the Lambda Kafka consumer group ID page), Kafka scaling modes (the Lambda scaling modes page), Kafka low latency (the Lambda Kafka low latency page), Queue mode (the Lambda Queue mode page), Kinesis (the Lambda Kinesis page), DynamoDB Streams (the Lambda DynamoDB Streams page), ESM (the ESM page), and metric types.
AWS Lambda API Reference, specifically the pages for CreateEventSourceMapping, SelfManagedKafkaEventSourceConfig, AmazonManagedKafkaEventSourceConfig, and EventSourceMappingMetricsConfig.
Five posts from the AWS Big Data Blog: an article on KCL 3.0 from November 6, 2024; an article on large consumer groups in MSK from September 22, 2026 (the Large Groups Blog); an article on the next-generation rebalance protocol in MSK from October 1, 2026 (the Next-Generation Protocol Blog); an article on cross-account configuration for KPL and KCL from October 12, 2026 (the Cross-Account Blog); and an article on building serverless streaming applications with MSK and Lambda (the MSK and Lambda article).
One post from the AWS Architecture Blog: an article on DynamoDB leases from September 10, 2026 (read only for Section 1.4).
Three entries from the AWS re:Post Knowledge Center: MSK consumer group rebalancing, MSK consumer lag, and KCL troubleshooting.
Documentation for Apache Kafka 4.2, specifically the Consumer Rebalance Protocol page, the consumer configuration page, the broker configuration page, and the Apache Kafka 4.2.0 release announcement.
The CHANGELOG for KCL on GitHub, and the source code for KCL 3.5.4 on Maven Central.
The Lambda service model for the AWS SDK for Python (botocore).

As of the verification date, the latest version of KCL was 3.5.4 (October 30, 2026).

1.4 Distinguishing Similar Terms

This article differentiates similar terms as follows:

Lease: In this article, "lease" refers to the record in the lease table that KCL uses to represent the association between a shard and a worker. The AWS Architecture Blog article from September 10, 2026, discussing DynamoDB leases, describes a custom mechanism that uses DynamoDB conditional writes to determine the owner of a WebSocket connection. It is not an article about KCL, and this article does not use it as a source for KCL.
Rebalance: In this article, "rebalance" refers to redistributing ownership of partitions within a group of Kafka consumers. The Intelligent Rebalancing feature in MSK Express, which moves partitions between brokers, is a separate function from consumer assignment. It is also distinct from EC2 spot instance rebalance recommendations.
Consumer: This article uses the term "consumer" to collectively refer to KCL workers, Kafka consumers, and Lambda ESMs. The "consumer" registered for Kinesis Data Streams enhanced fan-out is a registration that receives read bandwidth and is not a unit of assignment. The "consumer" in the Kafka consumer protocol refers to the protocol's name itself.
Shard Assignment: This article focuses on shard assignment from the consumer's perspective – determining which consumer reads which shard. Kinesis Data Streams' service-managed record distribution, added on September 23, 2026, is a feature where the service determines which shard a producer-written record is placed in. This is a separate layer from the consumer-side assignment.

2. How to Read the Tables

This section defines the columns in the tables from Sections 3 through 8, as well as how to describe the state of the sources. The column names and state terms are consistent across three articles within the same series.

2.1 Columns

The tables in this article are structured with a Consumer column at the beginning, followed by subsequent columns. In some tables, the first column may instead represent the type of change.

Who Decides: The entity responsible for determining which shard or partition belongs to which consumer.
Where the State Lives: The location where the decisions are recorded, and the position up to which records have been read.
What Changes It: The trigger that initiates a change in assignment.
What the Consumer Sees: What the consumer observes when the assignment changes – for example, stopping, duplication, or delays.
Where It Is Written: The source of the information in each row, specifying which page of which source.
No Statement Found: Matters for which no statement was found in the sources in Section 1.3, even after changing the search terms and the kind of source. When there is nothing, it is marked with —.

A non-empty No Statement Found entry does not indicate that the event is nonexistent or impossible.

2.2 Five States

The state of the sources is described using the following five terms:

No statement found: In the text, this is written as no statement was found, accompanied by the scope of the search.
Documentation does not list: The item is not present in the list or table. In the text, this is written as the documentation does not list.
Sources disagree: In the text, this is written as the sources disagree, and both statements, along with their sources, are presented. This article does not definitively identify either source as incorrect.
Inference: In the text, the same sentence states that it is this article's reading and not a statement in any source.
Implementation fact: This refers to information found in the implementation itself, such as in the KCL source code, and is not a documented promise. In the text, this is described as an implementation fact, along with the version and retrieval date.

2.3 Consumer Terminology

This article uses the following terms consistently:

worker: A KCL worker. This is a unit of processing within KCL that users execute, and each worker has one scheduler.
record processor: The user's processing code, created by a KCL worker for each shard it leases.
group coordinator: A role within a Kafka broker responsible for managing a consumer group.
group leader: In the classic protocol, the client selected to calculate assignments.
event poller: An AWS resource within Lambda's ESM that reads streams and topics.

This article does not use the same term to refer to Kafka consumers and consumers registered for Kinesis enhanced fan-out.

2.4 Strength of Sources

This article treats the sources as stronger in the following order:

Text from developer guides and API references.
SDK service models.
Apache Kafka documentation.
Text from AWS blogs and the re:Post Knowledge Center. In the text, blog posts and articles will be referenced with their names and dates within the same sentence.
KCL source code. It is written as an implementation fact (Section 2.2).

3. Three Places That Decide the Assignment

This section briefly lists the three methods for determining consumer assignment and presents them in a single table. Detailed explanations are provided in Sections 4 through 6.

3.1 KCL

In KCL, the workers that the user runs determine the shard owner through a DynamoDB lease table. The KCL concepts page defines a lease as follows:

Lease – defines the assignment between a worker and a shard. KCL consumer applications use leases to
distribute data record processing across multiple workers. Each shard is bound to only one worker by a lease
at any given time and each worker can hold one or more leases simultaneously.

The Kinesis Data Streams service, which provides streams, does not determine which worker reads which shard. Instead, KCL makes this determination; KCL operates within the user's process.

3.2 Kafka

In Kafka, the group coordinator within a broker manages the participants of a consumer group. The location where assignments are calculated varies depending on the protocol. The Next-Generation Protocol Blog states the following regarding the classic protocol:

The classic protocol relied on client-side rebalance logic with a global synchronization barrier.

The same blog post states the following about the consumer protocol:

KIP-848 shifts coordination logic from the client to the broker-side group coordinator.

3.3 Lambda Event Source Mappings

In Lambda's ESM, the AWS-side event pollers read from streams and topics. The ESM page describes an ESM as follows:

Event source mappings are Lambda resources created and managed within the Lambda service.

The Kafka ESM participates in a Kafka consumer group. The API reference description for ConsumerGroupId states:

The identifier for the Kafka consumer group to join.

However, no statement was found on who decides, and how, which poller holds which partition in the group. This is the no statement was found state, and the search covered the Lambda sources in Section 1.3.

3.4 Table by Layer

This section lists the three consumers with the columns in Section 2.1.

Consumer	Who Decides	Where the State Lives	What Changes It	What the Consumer Sees	Where It Is Written	No Statement Found
KCL 3.x	The user's worker. In the KCL 3.5.4 implementation, the leader worker calculates the assignments.	In this article's reading, the lease table in DynamoDB in the user's account. Starting with version 3.5, the single table format can be selected.	Worker increases/decreases and stops, or CPU utilization/throughput imbalance.	Moved shards stop until the new owner begins reading. Duplicates can occur during handoff.	KCL concepts page, KCL tables page, KCL configuration page, KCL migration page	A single sentence stating that the table resides within the user's account.
Kafka Classic Protocol	The group coordinator manages participants, and the group leader client calculates assignments.	The `__consumer_offsets` topic, in this article's reading (Section 3.5).	Consumer joins and leaves, topic changes.	According to re:Post and the Next-Generation Protocol Blog, all consumers stop. The scope of the stop depends on the assignment strategy.	Next-Generation Protocol Blog, Large Groups Blog, re:Post rebalancing article, Kafka 4.2 documentation	—
Kafka Consumer Protocol	The broker's group coordinator calculates assignments.	The `__consumer_offsets` topic, in this article's reading (Section 3.5).	Consumer joins and leaves, topic changes.	Only the affected partitions move, while other consumers continue processing.	Next-Generation Protocol Blog, Kafka 4.2 documentation	—
Lambda Kafka ESM	Not stated; AWS-side event pollers read. In Stream mode, each partition is assigned to a single consumer (stated for self-managed Kafka).	The Kafka consumer group's committed offsets.	Poller increases/decreases, another consumer with the same group ID.	Poller increases/decreases can cause rebalancing and latency.	Lambda Queue mode page, Lambda scaling modes page, Lambda Kafka low latency page, Lambda Kafka consumer group ID page	Whether Kafka or Lambda decides the assignment, a way to see which poller holds which partition, the default group ID
Lambda Kinesis & DynamoDB Streams ESM	Not stated; AWS-side event pollers read. A single batch contains only records from a single shard.	No statement was found on its location (Section 6.7). ESM is a resource managed in the Lambda service.	No statement was found (Section 6.7).	At-least-once processing; duplicates can occur.	Lambda Kinesis page, Lambda DynamoDB Streams page, ESM page	Location of the state, which poller holds which shard.

3.5 Locations Where State Persists

The assignment and the position up to which data has been read are stored in three locations:

KCL: Each row in the lease table represents a shard. leaseOwner indicates the owner, checkpoint is the position read, and leaseCounter reflects the owner's activity. Users can view this table in the DynamoDB console.
Kafka: The Large Groups Blog describes how, during rebalancing, the group coordinator writes GroupMetadata records to an internal topic called __consumer_offsets. The Kafka 4.2 broker configuration page lists the settings for the internal topic that receives offset commits (the offset commit topic). This article reads these two as the same topic.
Lambda: The MSK and Lambda article states that the Kafka ESM commits offsets after processing each batch. No statement was found on where the Kinesis and DynamoDB Streams ESMs keep the position (Section 6.7).

4. KCL — Assignment Recorded in Your Lease Table

This section describes how KCL 3.x determines the owner of a shard and where that information is stored. This article covers the assignment while the number of shards stays the same. Shard splitting and merging, and the resulting lease assignments for child shards, are covered in a separate article on resharding within the same series.

4.1 Tables: Three or One?

The KCL tables page describes how tables are structured in KCL 3.x as follows:

For each Amazon Kinesis Data Streams application, KCL 3.x creates three DynamoDB tables by default to manage
the metadata: lease table, worker metrics table, and coordinator state table. Starting with KCL 3.5, you can
use the single table format to consolidate all metadata into the lease table.

The roles of the three tables are as follows:

Lease Table: Contains rows for each shard. By default, KCL uses the application name as the table name.
Worker Metrics Table: Records the CPU utilization of each worker. The default name is the application name with -WorkerMetricStats appended.
Coordinator State Table: Contains data related to leader election and migration from 2.x to 3.x. The default name is the application name with -CoordinatorState appended.

In the single table format, the worker metrics and coordinator state information are distinguished by the entityType attribute and are stored in the lease table. The single table format page describes how to enable it for each current version. When migrating from KCL 2.x to 3.5, the single table format is the default. When migrating from KCL 3.0–3.4 to 3.5 or later, a two-stage deployment is required, and the default value for migrateAllEntitiesToLeaseTable is false. No statement was found on whether the tables will be one or three when KCL runs for the first time with 3.5 or later; the search covered the single table format page and the KCL tables page. This is the no statement was found state. As described in Section 8.2, in this configuration, KCL 3.5.4 created the table in the single table format from the start.

The KCL tables page also notes that the tables do not automatically disappear.

KCL consumer application does not automatically remove these three DynamoDB metadata tables.

The default capacity mode is on-demand.

4.2 The Lease Table Is in the User's Account

No single sentence was found explicitly stating that the lease table is created in the user's account. The search covered the KCL tables page, the KCL concepts page, the IAM permissions page, and the Cross-Account Blog. This article reads that the table resides in the account that runs the KCL, based on the following three statements:

The first is the billing sentence on the KCL tables page. This statement does not specify an amount, but rather identifies the account to which the table's expenses are charged.

Your account is charged for the costs associated with the DynamoDB table, in addition to the costs
associated with Kinesis Data Streams itself.

The IAM permissions page shows the format of the table's ARN (Amazon Resource Name). Permissions for the KCL are granted to tables with an ARN in the format arn:aws:dynamodb:region:account:table/KCLApplicationName.

The Cross-Account Blog describes the configuration. Even if the stream resides in a different account, the account running the KCL maintains its own lease table.

Account C (Consumer): Runs the KCL 3.5 application with its own DynamoDB lease table and CloudWatch metrics.

In this article's reading, the lease table is located in the account that runs the KCL, rather than within the account that owns the stream or within any AWS service. Because the table is the user's resource, it is the user's responsibility to delete the table and to grant permissions to it.

4.3 One Shard Is Held by Only One Worker

As stated on the KCL concepts page quoted in Section 3.1, at any given time, each shard is held by only one worker, although a single worker can hold multiple leases. Workers create a record processor for each shard that they hold a lease on. The KCL 1.x and 2.x page describes the behavior up to version 2.x as follows:

Every worker will contend to hold all available leases for all available shards in a data stream. But only
one worker will successfully hold each lease at any one time.

In KCL 3.x, the method of assignment has changed. The entry for version 3.0.0 (November 6, 2024) in the CHANGELOG states:

KCL 3.x introduces a new lease assignment and load balancing algorithm. It assigns leases among workers based
on worker utilization metrics and throughput on each lease, replacing the previous lease count-based lease
assignment algorithm.

In the implementation, the LeaseAssignmentManager in the KCL 3.5.4 source code (obtained from Maven Central on November 1, 2026) calculates the assignment only on the leader worker and writes the results to the lease table. Non-leader workers skip this calculation. The developer guide states that the coordinator state table contains data related to leader election, but no statement was found that the leader calculates the assignment; the search covered the KCL chapter of the developer guide.

4.4 Assignment Criteria — CPU, Throughput, and Lease Count

The KCL configuration page describes the assignment goals for KCL 3.x as follows:

In KCL 3.x, the load balancing algorithm aims to achieve even CPU utilization across workers, not an equal
number of leases per worker.

The KCL tables page describes the behavior when CPU utilization is unavailable.

If KCL cannot collect CPU utilization metrics from workers, KCL will fall back to using throughput per worker
to assign leases and balance the load across workers in the fleet.

The KCL migration page lists five conditions under which KCL can collect CPU utilization, by environment:

Amazon EC2: The operating system is Linux, and IMDSv2 is enabled on the instance.
Amazon ECS on Amazon EC2: The operating system is Linux, the task metadata endpoint v4 is enabled, and the container agent is version 1.39.0 or later.
Amazon ECS on AWS Fargate: The Fargate task metadata endpoint v4 is enabled, and the platform version is 1.4.0 or later.
Amazon EKS on Amazon EC2: The operating system is Linux.
Amazon EKS on AWS Fargate: The Fargate platform version is 1.3.0 or later.

This article reads that workers running outside of these environments, such as those on a developer's local machine, are assigned based on throughput.

The KCL concepts page describes KCL workers as follows:

KCL coordinates workers to consume data from the stream in a distributed manner and balances the load evenly
across multiple workers.

This article reads this even balancing as balancing CPU utilization or throughput, as described in the two preceding points. The even balancing here does not refer to the number of leases.

However, KCL also has a strategy for balancing based on the number of leases. In the implementation, the LeaseManagementConfig in KCL 3.5.4 has a default leaseAssignmentStrategy of WORKER_UTILIZATION_AWARE. Selecting LEASE_COUNT_BASED allows balancing based on the number of leases, without considering worker utilization. The CHANGELOG indicates that this strategy was added in version 3.4.3 (April 30, 2026). The KCL configuration page does not list this setting. This is the documentation does not list.

The KCL configuration page lists the following settings related to assignment.

reBalanceThresholdPercentage: Default 10. The percentage used to decide whether to reconsider the assignment.
dampeningPercentage: Default 60. The percentage by which to reduce the amount of work moved from workers experiencing high load during a single re-evaluation.
maxLeasesForWorker: Default is unlimited. The KCL configuration page notes that setting this value too low can make it difficult to balance load, and may result in data loss if all shards cannot be processed.
disableWorkerMetrics: Default is FALSE. If set to TRUE, CPU utilization will not be used in the assignment process.
maxLeasesToStealAtOneTime: KCL 3.x ignores this setting and assigns leases based on the resource utilization of workers.

4.5 Leases of a Failed Worker

If a worker fails to renew its lease for a certain period, KCL will consider that worker to have failed. The KCL configuration page specifies failoverTimeMillis, which defaults to 10,000 milliseconds, or 10 seconds.

The number of milliseconds that must pass before you can consider a lease owner to have failed.

The KCL concepts page describes that when a worker stops or fails, losing its lease, KCL will assign that lease to another worker. The KCL configuration page also states that KCL 3.x distributes expired leases among workers.

The time it takes for another worker to begin processing after a worker fails is not solely determined by failoverTimeMillis. In the implementation, the assignment calculation in KCL 3.5.4 operates at a regular interval. If leaseAssignmentIntervalMillis is not configured, the interval defaults to 2 * failoverTimeMillis, which is 20 seconds. The DynamoDB KCL migration page states that the time before processing begins for newly discovered shards defaults to 1.5 × failoverTimeMillis when leaseAssignmentIntervalMillis is not set. This article reads the two as statements about different quantities and does not treat them as a disagreement about one value.

The new owner will begin reading from the checkpoint that the failed worker last recorded. The KCL concepts page explains that because of checkpoints, workers can continue processing from the point where the previous worker stopped. This article reads that if the failed worker had already processed records after the checkpoint, those records will be processed a second time.

4.6 Graceful Lease Handoff and the Duplicates That Remain

KCL 3.x uses graceful lease handoff when reassigning a lease to a different worker. The KCL migration page describes this feature as:

This is achieved by checkpointing the last processed sequence number in the lease table before the lease
handoff. To ensure the graceful lease handoff works properly, you must make sure that you invoke the
checkpointer object within the shutdownRequested method in your RecordProcessor class.

The same page also notes that duplicates can remain.

While graceful lease handoff significantly reduces the likelihood of data reprocessing during lease
transfers, it does not entirely eliminate this possibility.

The KCL configuration page sets the default value of isGracefulLeaseHandoffEnabled to TRUE and the default value of gracefulLeaseHandoffTimeoutMillis to 30,000 milliseconds, or 30 seconds. This 30-second timeframe is the minimum time KCL waits for the current record processor to shut down; if this time elapses, the lease is forcibly transferred to the next owner.

The KCL migration page describes this feature in the context of reassigning a lease to a different worker during a reassignment. This article reads that if a worker's process is forcibly terminated, the shutdownRequested method is not called, triggering the failover path described in Section 4.5. In the configuration outlined in Section 8.5, even leases from workers that were shut down gracefully were eventually transferred to another worker after their expiration. In either path, design the downstream system to accept duplicates. The KCL migration page recommends building the downstream system to be idempotent.

4.7 Migrating from 2.x to 3.x

KCL 2.x and 3.x use different methods for determining assignments. The KCL migration page specifies that workers for the same application must use the same assignment method at the same time.

It's important that all workers in your consumer application use the same load balancing algorithm at a
given time because KCL 2.x and 3.x use different load balancing algorithms.

Therefore, the migration process is performed in two phases. First, set the ClientVersionConfig to CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1 and distribute it to all workers. Then, change it to CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X and begin the migration. The KCL migration page states how long the switch to the new method takes after all workers are ready. The original spelling of leasee is retained in the quotation.

It takes at least 10 minutes for KCL to switch to the new leasee assignment algorithm after all workers are
ready to run it.

To verify that the migration is complete, check if the sum of the CurrentState:3xWorker metric in CloudWatch matches the number of workers. The rollback page states that it is not possible to roll back directly from Phase 2 to 2.x; instead, you must roll back from Phase 2 to Phase 1, and then from Phase 1 to 2.x.

4.8 The Phase of KCL 1.x

The table on the KCL lifecycle page, as of November 1, 2026, lists the KCL 1.x row as follows. The column names remain consistent with the headings on the page.

Major version	Current phase	Release date	Maintenance mode date	End-of-support date
KCL 1.x	`Maintenance mode`	2013-12-19	2025-04-17	2026-01-30

The Current phase column remains Maintenance mode and the end-of-support date of January 30, 2026, has passed. The KCL 1.x and 2.x pages, as well as the Adapter page, still use future tense language.

KCL 1.x will reach end-of-support on January 30, 2026.

This article does not state that KCL 1.x has ended support, nor that it is still in maintenance mode. It simply presents the two columns in the table and the future tense language as they are. The KCL 2.x and 3.x versions are listed as General availability in the table.

4.9 DynamoDB Streams Kinesis Adapter

When reading DynamoDB Streams with KCL, use the DynamoDB Streams Kinesis Adapter. The assignment mechanism is that of KCL. The Adapter page recommends KCL 3.5.x or later with AWS SDK for Java v2. The DynamoDB KCL migration page states that KCL 3.5.x or later is supported with Adapter 2.4.x or later. Check the version combination against both pages.

4.10 How to Check

KCL assignments can be verified by examining the rows in the lease table. The KCL tables page describes the attributes of each row as follows:

leaseKey: The shard ID when reading a single stream.
leaseOwner: The worker currently holding the lease.
checkpoint: The sequence number of the last checkpoint performed on that shard.
leaseCounter: A value used to verify whether a worker is processing the lease. This value increases when the lease is transferred to a different worker.
ownerSwitchesSinceCheckpoint: The number of times the lease owner has changed since the last checkpoint.

The re:Post KCL troubleshooting article describes reading the table this way: if the checkpoint is not updated, the processRecords process may have stopped. Furthermore, if neither the checkpoint nor the leaseCounter are updated, the maxLeasesPerWorker=1 parameter prevents other workers from acquiring the lease.

5. Kafka — Rebalancing Managed by the Group Coordinator

This section outlines how the partition ownership is determined within Kafka consumer groups, comparing the classic and consumer protocols. This section is based solely on documentation and does not include demonstrations using MSK or self-managed Kafka.

5.1 Group Coordinator

For each consumer group, one of the brokers will act as the group coordinator. The re:Post article on MSK rebalancing states that when a consumer no longer reaches a cluster, the following occurs.

If a consumer can't reach the cluster, then the group coordinator must remove the consumer from the
consumer group.

The same article explains that when a consumer is removed, a rebalancing process begins. MSK revokes the partitions assigned to the remaining consumers, and the group coordinator redistributes the topic partitions to the remaining consumers.

5.2 Conditions for Rebalancing

The re:Post article on MSK rebalancing lists five conditions that trigger a rebalance.

A partition ownership is reassigned to a different consumer within the consumer group.
A new consumer is added to the consumer group.
A consumer stops, fails, or leaves the group.
The topic is modified, resulting in a partition reassignment.
The topics being consumed by the group do not match the topics assigned to each consumer.

5.3 Classic Protocol

In the classic protocol, the assignment is computed on the client side. The Large Groups Blog notes the difference from KIP-848:

The broker now computes partition assignments server-side rather than delegating to a consumer group leader.

The Next-Generation Protocol Blog describes the classic protocol as follows:

Every rebalance caused all consumers in the group to pause processing simultaneously regardless of whether
their partition assignments were changing.

The re:Post article on MSK consumer lag also states that all assignments are revoked during a rebalance.

During a consumer group rebalance, all the consumer partition assignments are revoked.

However, the classic protocol also includes cooperative rebalancing. The Next-Generation Protocol Blog itself describes CooperativeStickyAssignor as a client-side assignment strategy that supports incremental and cooperative rebalancing. The blog also includes a table illustrating the impact on consumers of the classic protocol, depending on the assignment strategy employed. The Kafka 4.2 consumer configuration page indicates that the default for partition.assignment.strategy is set to both RangeAssignor and CooperativeStickyAssignor, with RangeAssignor used by default. This article does not settle on one scope of what pauses in the classic protocol. This article reads that the re:Post sentence applies to assignment strategies that do not revoke incrementally, and that the scope of the pause is smaller with CooperativeStickyAssignor. No statement was found on how much each strategy pauses; the search covered the Next-Generation Protocol Blog, the re:Post rebalancing article, and the Kafka 4.2 documentation.

According to the Next-Generation Protocol Blog, the classic protocol shows up in the broker logs as the PreparingRebalance state and the line Assignment received from leader.

5.4 Consumer Protocol (KIP-848)

The KIP-848 consumer protocol calculates assignments on the broker side. The Kafka 4.2 Consumer Rebalance Protocol documentation states that this protocol reached General Availability (GA) in Kafka 4.0. The Next-Generation Protocol Blog describes the process as follows:

Only affected partitions move during a rebalance, and consumers with unchanged assignments continue
processing uninterrupted.

With this protocol, the server side configures the heartbeat interval, session timeout, and assignment strategy. These server-side configurations are controlled by group.consumer.heartbeat.interval.ms, group.consumer.session.timeout.ms, and group.consumer.assignors. The Kafka 4.2 Consumer Rebalance Protocol documentation specifies that the default assignment strategies for group.consumer.assignors are uniform and range. The client-side configurations heartbeat.interval.ms, session.timeout.ms, and partition.assignment.strategy are not supported by this protocol.

Existing consumer groups can be migrated without interruption. The Kafka 4.2 Consumer Rebalance Protocol documentation indicates that when consumers with group.protocol=consumer are rolled out one by one, the group is converted from Classic to Consumer when the first such consumer joins. However, this only applies when the client-side assignment strategy does not embed its own metadata. The same documentation lists the inability to use client-side assignment strategies as a limitation.

5.5 The Defaults Differ Between the Server Side and the Client Side

Kafka 4.0 and later versions have different default settings for the consumer protocol on the server and client sides. The Kafka 4.2 documentation on the Consumer Rebalance Protocol states the following regarding the server side:

The new consumer protocol is automatically enabled on the server since Apache Kafka 4.0.

The same documentation states the following regarding the client side:

However, the protocol is not enabled by default. The group.protocol configuration must be set to consumer to
enable it.

The Kafka 4.2 documentation on consumer configuration sets the default value of group.protocol to classic. The Kafka 4.2 documentation on broker configuration sets the default value of group.coordinator.rebalance.protocols to classic,consumer,streams.

Two AWS blog posts address this default in different ways. The Large Groups Blog states:

Apache Kafka 4.0 (released March 2025) adopted KIP-848 as the default consumer protocol.

The Next-Generation Protocol Blog writes the following in its client configuration step, and its configuration example carries the comment defaults to 'classic' if omitted.

Set group.protocol=consumer in your consumer configuration. The protocol is not enabled by default:

This is the sources disagree. This article reads the wording in the Large Groups Blog to refer to the default behavior on the server side. However, the blog post itself does not explicitly state this. In this article's reading, simply upgrading Kafka to version 4.x will not change the protocol for clients that do not explicitly specify group.protocol. The Next-Generation Protocol Blog states that even if a client's library does not support KIP-848, it will still default to the classic protocol.

5.6 MSK Standard and Express Brokers

The Next-Generation Protocol Blog describes support for MSK as follows:

You can use the consumer protocol on Amazon MSK on all 4.x Apache Kafka versions on both MSK Standard and
Express brokers.

The MSK versions page lists the new consumer rebalance protocol of Kafka 4.0 as generally available. Server-side configuration handling differs based on the broker type. The Next-Generation Protocol Blog states:

In Amazon MSK Express brokers, these configurations are read-only and cannot be modified.

For Standard brokers, the same blog post states that these settings are managed with the broker configuration. On the Express configuration page, the read-only settings table does not include any settings that begin with group.consumer. This is the documentation does not list, and the statement that these settings are read-only on Express brokers is only found in the Next-Generation Protocol Blog.

The MSK versions page states that, as of November 1, 2026, Kafka 4.2 is available only on Express brokers.

Apache Kafka version 4.2 is currently available on MSK Express brokers only.

5.7 Static Membership and Session Timeout

Static membership can be used to prevent rebalancing when consumers briefly disconnect and then reconnect. To do this, configure the consumer's settings with a fixed string that identifies the consumer in the group, using the group.instance.id setting. The troubleshooting page for MSK states the following regarding static membership:

Using Static Membership Protocol only reduces the probability of encountering this issue.

That section of the page covers KAFKA-9752 in Kafka 2.3.1 and 2.4.1 and gives the default client session timeout as 10 seconds. The consumer configuration page for Kafka 4.2 sets the default value for session.timeout.ms to 45,000 milliseconds, or 45 seconds. The broker configuration page for Kafka 4.2 also sets the default value for group.consumer.session.timeout.ms to 45 seconds. This article places the 10 seconds, from a section on old versions, side by side with the 45 seconds for Kafka 4.2.

The Next-Generation Protocol Blog states that if a consumer with static membership terminates without calling consumer.close(), the group coordinator will wait for the entire session timeout before reassigning partitions. The same blog post also notes that it is not possible to modify group.consumer.session.timeout.ms on Express brokers.

5.8 Large Groups and the GroupMetadata Record

The Large Groups Blog describes the limit for large consumer groups as follows. The blog itself states that its procedures apply to Standard brokers in MSK Provisioned clusters.

During a rebalance, the Group Coordinator serializes a GroupMetadata record containing information about
every member in the group and persists it to the __consumer_offsets topic.

The blog post also notes that if this record exceeds the default 1 MB, it can lead to RecordTooLargeException errors and repeated rebalancing. The same blog post explains that, with the consumer protocol, each member no longer carries all subscription and assignment data, so the metadata per member becomes smaller.

5.9 Share Groups

A share group is a mechanism that allows multiple consumers to read a partition simultaneously, added by KIP-932, Queues for Kafka. The Lambda Queue mode page contrasts share groups with traditional consumer groups, which exclusively assign partitions to consumers. The announcement for the Apache Kafka 4.2.0 release states that Kafka Queues (Share Groups) are now production-ready.

As of November 1, 2026, no statement was found of a configuration in which share groups can be used on MSK; the search covered the MSK versions page and the Lambda Queue mode page. The MSK versions page states that Queues are a preview feature in Kafka 4.1 and that Kafka 4.2 is available only on Express brokers. Further, it states:

KIP-932: Queues for Kafka is not yet supported on MSK Express brokers.

This article does not state that share groups cannot be used with MSK. Instead, it writes that, as of November 1, 2026, no statement was found in the MSK sources of a configuration that uses share groups in production.

5.10 How to Check

The Next-Generation Protocol Blog outlines the following verification methods:

Use kafka-consumer-groups.sh --describe to check if all members are operational and stable.
Examine the broker logs to determine which protocol is running. With the consumer protocol, the term fenced and a line in the form of the first line below appear. With the classic protocol, the state PreparingRebalance and a line in the form of the second line below appear.
Using the CloudWatch metrics SumOffsetLag and EstimatedMaxTimeLag, verify that the lag returns to zero after rebalancing.

The two example log lines in the Next-Generation Protocol Blog are quoted as is:

[GroupCoordinator id=X] [GroupId <group-id>] Member <member-id> joins the consumer group using the consumer protocol.
[GroupCoordinator id=X] Assignment received from leader for group <group-id>

6. Lambda Event Source Mappings — Read by AWS-Side Event Pollers

This section separates what the sources write about assignment in Lambda ESMs from what they do not. This article did not try the Kafka ESM in the demo. The Kafka rows are based solely on the text of the sources.

6.1 An ESM Is a Lambda Resource

The ESM page describes an ESM as a resource created and managed within the Lambda service. Regarding Kinesis ESM, the Lambda Kinesis page describes the function as an application that consumes streams.

Your Lambda Event Function is a consumer application for your data stream. It processes one batch of records
at a time from each shard.

No statement was found on whether Lambda's ESM uses KCL or creates tables in the user's DynamoDB. This is the no statement was found state, and the search covered the Lambda sources in Section 1.3.

6.2 Kafka's Stream Mode

The page describing Lambda's Queue mode, when explaining the two modes of self-managed Kafka's ESM, describes Stream mode as follows:

Stream (default). Uses Kafka consumer groups with per-partition ordering. Lambda assigns each partition to
one consumer.

The comparison table on the same page describes the consumer model of Stream mode as a consumer group with exclusive partition assignment. This description appears on the page regarding self-managed Kafka. No sentence of the same form was found for the MSK ESM in the sources that this article read. For the MSK ESM, those sources address the cap on the number of pollers (Section 6.3) and the handling of group IDs (Section 6.4).

No statement was found on whether the Kafka group coordinator or Lambda itself makes this assignment. This is the no statement was found state, and the search covered the Lambda sources in Section 1.3 and the MSK and Lambda article.

6.3 Number of Pollers and Rebalancing

The Lambda scaling modes page states that Lambda caps the maximum number of pollers in provisioned mode at the number of partitions.

In addition, to maintain ordered processing within partitions, Lambda caps the MaximumPollers to the number of
partitions in the topic.

The Lambda Kafka low latency page indicates that automatic increases and decreases in the number of pollers can trigger a rebalancing process.

Provisioned mode automatically scales your event pollers based on traffic patterns within configured minimum
and maximum event pollers which can trigger rebalance, and therefore, introduce additional latency.

The same page also states that setting the minimum and maximum number of pollers to the same value can prevent these automatic adjustments. The configuration of the number of pollers itself is covered by this site's Lambda Event Pollers in Provisioned Mode for Kafka and SQS.

6.4 Sharing a Group ID Means Receiving Only a Portion

The Lambda Kafka consumer group ID page states that if other consumers are using the group ID specified for the ESM, it will behave as follows:

Kafka distributes messages across all consumers in a consumer group. If you specify a consumer group ID that
has other active consumers, Lambda receives only a portion of the messages from the Kafka topic.

The same page explains that if a valid group with the same ID exists, Lambda will disregard the StartingPosition specified in ESM and instead begin reading from the group's committed offset. The specified group ID must be unique across all Kafka event sources, and cannot be changed after it is created. No statement was found on the default group ID used when no ID is specified. This is the no statement was found state, and the search covered the Lambda sources in Section 1.3.

6.5 Kafka Queue Mode

Queue mode reads with a share group. The Lambda Queue mode page limits it to self-managed Kafka.

Self-managed Apache Kafka (SMK) clusters only. Amazon Managed Streaming for Apache Kafka (Amazon MSK) does not
currently support Queue mode.

The same page indicates that this mode can be used on clusters that run Kafka 4.2 or later with share groups enabled. However, it notes that order is not guaranteed and that the StartingPosition parameter cannot be used. The mode is configured using the ConsumptionMode setting. Both the Lambda Queue mode page and the botocore Lambda service model include ConsumptionMode in the SelfManagedKafkaEventSourceConfig, with values of Stream and Queue, defaulting to Stream. As of November 1, 2026, the Lambda API Reference documentation for SelfManagedKafkaEventSourceConfig and CreateEventSourceMapping does not list the ConsumptionMode setting. This is the documentation does not list. Similarly, AmazonManagedKafkaEventSourceConfig does not mention ConsumptionMode in the API Reference or in botocore. No What's New announcement was found for the date when Queue mode became available.

6.6 Kinesis and DynamoDB Streams

The Lambda Kinesis page states that one batch holds records from only one shard.

Each batch contains records from a single shard/data stream.

The same page indicates that you can specify a ParallelizationFactor from 1 to 10 to determine the number of batches that can read from a single shard simultaneously. The default value is 1, and increasing this number maintains order based on the partition key. Using the standard iterator, Lambda reads each shard at a base frequency of once per second, sharing read capacity with other consumers of the same shard. The Lambda DynamoDB Streams page states that Lambda reads shards from a DynamoDB stream at a base frequency of four times per second.

The Lambda Kinesis page also describes how it handles duplicates.

Lambda event source mappings process each event at least once, and duplicate processing of records can occur.

6.7 What Is Not Written

Regarding Lambda's ESM, no statement was found on the following. This is the no statement was found state, and the search covered the Lambda sources in Section 1.3.

In Kafka's Stream mode, how is it determined which pollers are assigned to which partitions?
Where do Kinesis and DynamoDB Streams' ESM store the state indicating how much of each shard has been read?
What mechanisms are available for users to view which pollers are assigned to which partitions or shards?
What is the default group ID used in Kafka's ESM when a group ID is not specified?
For Kinesis and DynamoDB Streams, how does the ESM change which poller reads which shard when the number of pollers changes or a poller fails?

6.8 How to Check

In Lambda's ESM, the assignment itself is not visible. Only the results are visible.

For Kinesis and DynamoDB Streams, you can observe latency using the IteratorAge metric. The metric types page describes IteratorAge as the time elapsed between a stream receiving a record and ESM forwarding that record to the function.
For Kafka, you can observe latency using the OffsetLag metric. The metric types page describes OffsetLag as the difference between the offset of the last record written to the topic and the offset of the last record processed by the function's consumer group. Enabling KafkaMetrics in the EventSourceMappingMetricsConfig also allows you to receive MaxOffsetLag and SumOffsetLag.
This article reads that the assignment of a Kafka group can be seen with kafka-consumer-groups.sh --describe if the group ID is known. As Section 6.7 states, no statement was found on the default ID, so where to find the group ID of an ESM is not stated.

7. What a Consumer Sees When the Group Changes

This section lists four changes, duplicates, and how to check, in the columns of four consumers. The Lambda column combines the Kafka, Kinesis, and DynamoDB Streams ESMs. The cells in the table summarize the statements and readings in Sections 4 through 6.

Change	KCL 3.x	Kafka Classic Protocol	Kafka Consumer Protocol	Lambda ESM
Increasing Consumers	In the KCL 3.5.4 implementation, on the next assignment run, the leader worker moves leases from heavily loaded workers. The transfer uses graceful lease handoff.	Full group rebalancing. The scope of the interruption depends on the assignment strategy.	Only the affected partitions move; the other consumers continue processing.	Increasing the number of pollers in provisioned mode can trigger a rebalance. No statement was found for Kinesis and DynamoDB Streams (Section 6.7).
Consumer Failure	After `failoverTimeMillis`, in the KCL 3.5.4 implementation, the next assignment calculation transfers to a different worker. It restarts reading from the last `checkpoint`.	After a session timeout, the group coordinator removes the consumer and initiates a rebalance.	After a session timeout, the consumer is removed, and only its partitions move.	No statement was found (Section 6.7).
Version Upgrade	Upgrading from 2.x to 3.x requires a two-stage process, and the switchover takes at least 10 minutes.	When consumers with `group.protocol` set to `consumer` join, the group is converted to a Consumer group.	—	—
Sharing Group and App Names	Workers with the same app name share the same lease table and split the shards. The KCL tables page says to keep the app name unique.	Consumers with the same group ID share partitions.	Consumers with the same group ID share partitions.	Kafka ESM only receives a portion of the messages from a topic.
Duplication	Duplication can occur during both lease handoffs and failovers.	In this article's reading, records after the last commit can be read again.	In this article's reading, records after the last commit can be read again.	At-least-once processing; duplicates can occur.
How to Check	Check the lease table's `leaseOwner`, `checkpoint`, and `leaseCounter`.	Use `kafka-consumer-groups.sh --describe` and broker logs.	Use `kafka-consumer-groups.sh --describe` and broker logs.	Check `IteratorAge` and `OffsetLag`.

The duplicates row for Kafka is this article's reading. After a rebalance, the new owner reads again from the offset after the last commit made before the rebalance. No direct statement on this was found in the AWS sources that this article read.

What a Consumer Sees When the Group Changes

8. What This Setup Showed

This section describes what was observed while running KCL workers and a Lambda ESM within this specific configuration. The observations are records of two runs of this setup and do not represent a performance comparison. If there are discrepancies between the documented material and the actual observations, both will be presented, and the observations will not override the documented material.

8.1 Setup

A single provisioned Kinesis data stream with four shards, located in the us-east-1 Region.
A single producer that writes records with sequential IDs, at a target rate of 20 records per second, using 64 partition keys.
KCL 3.5.4 workers, running on a developer's local machine, outside the prerequisites for worker metrics. They use polling for reading and do not use enhanced fan-out. The record processor checkpoints after processing each batch and also checkpoints upon receiving a shutdownRequested signal.
One Lambda Kinesis ESM, configured with a starting position of LATEST.
The lease table is read every three seconds and recorded with timestamps.
The test involves sequentially starting three workers. One worker is gracefully shut down (SIGTERM), one is forcibly terminated (SIGKILL), and a new worker is added. Each test run lasts approximately 22 minutes.
The test was executed twice on November 1, 2026. The first run occurred from 10:46 AM to 11:07 AM (UTC), and the second run occurred from 11:13 AM to 11:35 AM (UTC). During the second run, the table from the first run was left intact, and the test was executed with a different worker ID.

This article masks the account ID, ARN, and names of the stream and table. The table name is represented as <app>. Timestamps are provided as seconds elapsed from the start of each run.

8.2 Tables Created

KCL created only one table. The table was named after the application, while tables named -WorkerMetricStats and -CoordinatorState were not created. The items in the table were divided into four types based on the entityType value:

LEASE: Lease information for each shard, including fields such as leaseOwner, checkpoint, leaseCounter, and ownerSwitchesSinceCheckpoint.
LEADER: Information related to leader election.
WORKER_METRIC_STATS: Metrics for each worker.
CLIENT_VERSION_MIGRATION: Information about client version migration status.

The worker logs showed the following line (table name masked):

Legacy WorkerMetricStats table <app>-WorkerMetricStats does not exist, delegate not enabled

As Section 4.1 states, no statement was found describing the number of tables created when running KCL for the first time on version 3.5 or later. In this configuration, KCL 3.5.4 created the table in the single table format from the start.

In this setup, Lambda's ESM did not create any DynamoDB tables within the user's account. In the first run, when comparing the list of tables before the stack was created with the list of tables before the KCL workers were launched (while ESM was running, 90 seconds after the scenario began), no new tables were found. This simply indicates that no tables were visible in this configuration; it does not prove that ESM does not create tables.

Even after all workers were stopped, the table remained. The table persisted until the user manually deleted it at the end of the process.

8.3 Waiting for GSI Availability

During the first run, for approximately 16.5 minutes after the initial worker started, no leases were assigned to any of the workers. During this period, the worker logs displayed the following line, appearing every 10 seconds:

DynamoDBLeaseRefresher - GSI status is not active, trying again in 10s

This line continued from t=114 seconds to t=1106 seconds. Then, at t=1116 seconds, the process of assigning leases began, and at t=1128 seconds, leases were created for four shards. Consequently, during this first run, any workers that joined and stopped during this interval processed no records at all.

The KCL tables page states that KCL creates a GSI on the lease table with leaseOwner as the partition key. The CHANGELOG entry for version 3.5.1 states:

Extend GSI creation timeout to 1 hour

The length of this wait time was observed once in this configuration. This article reads that when a new KCL application is launched for the first time, no assignments will begin until the GSI becomes available. The second run, which started with the same table, did not show this wait.

8.4 Assignment Criteria and the Number of Leases

The worker metrics items had empty metric values. The worker logs produced the following two lines:

Unable to retrieve instance metadata
Worker [worker-r2w1] is using default WorkerMetricStats, setting initial utilization ratio to [1.0].

This article reads that the workers in this setup could not report CPU utilization, and KCL assigned leases with default values. As described in Section 4.4, the KCL tables page states that KCL then assigns leases based on throughput.

In the second run, the number of leases changed as follows:

When there was one worker, it held all four leases.
The second worker received its first lease 41 seconds after joining, and received a second lease 60 seconds later. The third worker received its first lease 47 seconds after joining. Leases moved one at a time.
When there were two workers, there was a period when the leases were split one and three (from t=714 seconds to t=876 seconds).

The number of leases was not always evened out across the workers.

8.5 Handoff When a Worker Stops

In the second run, this article measured the time until the lease ownership changed, and the time it took for the new owner to process the initial record, for each method of stopping the worker.

Change	Worker exited	Owner changed in the lease table	New owner's first record
Graceful stop (SIGTERM)	6 seconds later	49 seconds later	52 seconds later
Forced termination (SIGKILL)	Immediately	31 seconds later (2 shards), 51 seconds later (1 shard)	41 seconds later (2 shards), 61 seconds later (1 shard)

The gracefully stopped worker exited 6 seconds after SIGTERM, but the lease table still listed it as the owner for approximately 43 seconds afterward. This article reads that, even when a worker is stopped gracefully, its leases are not passed to another worker immediately; they are passed on an assignment run after they expire. As mentioned in Section 4.6, the KCL migration page writes about graceful lease handoff for the case where a lease is passed during a reassignment.

When a lease is moved during a reassignment, the record processor of the original owner recorded both the shutdownRequested checkpoint and the leaseLost event, in that order. For the first worker of the second run, this pair of events was recorded 7 times. Of those, 5 times it occurred during a reassignment when the lease was moved, and 2 times it occurred when the worker itself was stopped gracefully. Even for a gracefully stopped worker, the lease table still listed that worker as the owner after this pair of events was recorded.

In both cases – whether the worker was stopped gracefully or terminated forcibly – the time it took for the new owner to begin processing data was longer than the sum of the default 10 seconds of failoverTimeMillis and the 20-second interval of the assignment run in the KCL 3.5.4 implementation.

8.6 Duplication and Omission

In the second run, none of the 6,790 records written by the producer were processed more than once by KCL workers. The 299 records that were not processed were all written from t=1203.9 seconds on, and the workers began to stop at t=1205 seconds.

This article reads that the absence of duplicates is consistent with the record processor in this setup checkpointing after each batch. The KCL migration page states that graceful lease handoff does not eliminate the possibility of duplicates. The fact that no duplicates were observed in this particular run does not contradict this statement.

8.7 leaseCounter Behavior

The leaseCounter continued to increase even while the owner remained unchanged. The value, which was 15 to 55 at the start of the second run, reached a maximum of 754 approximately 19 minutes later. The KCL tables page describes leaseCounter as follows:

leaseCounter: Used for checking if a worker is currently processing the lease actively. leaseCounter increases
if the lease ownership is transferred to another worker.

This article reads that the leaseCounter in this configuration increased not only when the owner changed, but also each time a worker renewed the lease. This article sets the source sentence and the observation side by side and does not rewrite the source sentence with the observation. To see whether the owner has changed, look at the value of leaseOwner, not at how leaseCounter increases.

8.8 What This Setup Does Not Show

This article did not try MSK or self-managed Kafka. The content related to Kafka in Section 5 is based solely on the sources.
There were two runs, and the times are values from this setup.
The path that assigns leases by CPU utilization was not observed, because the workers did not run in an environment that meets the prerequisites for worker metrics.
This setup does not show where Lambda's ESM stores its state. It only showed that no new table appeared in the user's account.

9. Where the Sources Disagree and What They Do Not State

This section consolidates instances of conflicting information, missing details, and absent passages identified throughout Sections 4 through 6.

9.1 Where the Sources Disagree

Kafka 4.0 Consumer Protocol Defaults: The Large Groups Blog states that KIP-848 became the default. However, the Next-Generation Protocol Blog and the Kafka 4.2 documentation state that it is not enabled by default on the client. This article reads this as the difference between the client-side and server-side defaults (Section 5.5).
The phase of KCL 1.x: The table on the KCL lifecycle page lists the current phase as Maintenance mode and the end-of-support date as January 30, 2026, which has passed. The KCL 1.x and 2.x pages, as well as the Adapter page, contain a sentence in the future tense (Section 4.8).
Session Timeout Defaults: A troubleshooting page for MSK states a 10-second timeout for Kafka 2.3.1 and 2.4.1. The Kafka 4.2 documentation, however, states that the client and server both default to 45 seconds. The two statements are about different versions (Section 5.7).
Scope of Stoppage with the Classic Protocol: The re:Post article on consumer lag states that all assignments are revoked. The Next-Generation Protocol Blog makes it depend on the assignment strategy, and the Kafka 4.2 documentation lists a cooperative assignment strategy (Section 5.3).
What evenly means in KCL: The KCL concepts page states that it distributes load evenly. The KCL configuration page, however, states that it does not distribute leases evenly. This article reads this as the difference between even load distribution and even lease distribution (Section 4.4).

9.2 What the Documentation Does Not List

The configuration for KCL's leaseAssignmentStrategy. It appears in the implementation and the CHANGELOG, but not on the KCL configuration page (Section 4.4).
The ConsumptionMode in the Lambda API Reference. This is documented on the Lambda Queue mode page and in botocore, but not in the API Reference page (Section 6.5).
The configuration settings that begin with group.consumer on the Express configuration page. The statement that these settings are read-only is found in the Next-Generation Protocol Blog (Section 5.6).

9.3 Where No Statement Was Found

A statement, in one sentence, confirming that the lease table exists within a user's account. This article derived it from three statements (Section 4.2).
The number of tables used when running KCL for the first time with version 3.5 or later (Section 4.1). In this configuration, there was one (Section 8.2).
A statement that the leader worker calculates the assignment in KCL 3.x. The implementation has it (Section 4.3).
For Lambda's Kafka ESM, details regarding who determines assignments, the default group ID, and how to view the mapping between pollers and partitions (Sections 6.2 and 6.7).
The location where Lambda's Kinesis and DynamoDB Streams ESM stores its state (Section 6.7).
A configuration in which share groups can be used in production on MSK (Section 5.9).
The date of a What's New announcement for Lambda's Queue mode (Section 6.5).

10. Failure Modes

This section outlines failures that can occur when the layer that decides the assignment is mistaken, drawing upon the content of Sections 4 through 7.

Choosing the number of workers as a divisor of the number of shards, on the assumption that the leases will then be distributed evenly. KCL 3.x defaults to aligning based on CPU utilization or throughput, which may result in an uneven distribution of leases per worker.
Setting maxLeasesForWorker too low in an attempt to equalize the number of leases can lead to load imbalances, potentially preventing all shards from being processed and resulting in data loss, as noted on the KCL configuration page.
Not designing downstream components to be idempotent, on the assumption that graceful lease handoff prevents duplicates. The KCL migration page states that the possibility of duplicates does not go away.
Failing to perform checkpoints using shutdownRequested means, in this article's reading, that the final position is not recorded and the next owner begins reading from an older position.
After decommissioning a KCL consumer, metadata tables are left behind. KCL does not automatically delete these tables, and, in this article's reading, they remain in the user's account.
The same application name is used for KCL applications with different purposes. Since both applications share the same lease table, they effectively share shards.
Upgrading an MSK cluster to Kafka 4.x is assumed to switch to the consumer protocol. If the client does not specify group.protocol, it will continue to use the classic protocol.
Trying to raise group.consumer.session.timeout.ms on Express brokers so that static membership consumers have time to restart. The Next-Generation Protocol Blog states that this setting cannot be modified on Express.
A Lambda function's Kafka ESM group ID is configured with the group ID of other consumers. As a result, the Lambda function only receives a portion of the topic's messages.
Looking for the shard assignment of a Lambda ESM in the user's DynamoDB, like a KCL lease table. No statement was found that the ESM creates such a table (Section 6.1).

11. Frequently Asked Questions About Stream Consumer Assignment

This section addresses common questions that readers of Section 1.1 may have, drawing on the content of Sections 4 through 7. The answers are based on the sources named in each section.

Q1. Does adding KCL workers distribute the shards evenly?

Not necessarily by the number of leases. The default assignment in KCL 3.x aims to balance the CPU utilization across workers, and if CPU utilization cannot be collected, it will attempt to balance based on throughput. For distributing shards by the number of leases, the KCL 3.5.4 implementation and the CHANGELOG show LEASE_COUNT_BASED for the leaseAssignmentStrategy setting from KCL 3.4.3. However, this setting is not listed on the KCL configuration page (Section 4.4).

Q2. If a KCL worker fails, how long until another worker starts reading?

After failoverTimeMillis, 10 seconds by default, a worker is considered to have failed. In the implementation, the leader worker in KCL 3.5.4 calculates the assignment every 20 seconds by default. This article reads that the time until another worker starts reading includes both of these intervals. In the configuration described in Section 8.5, it took 41 to 61 seconds.

Q3. Does graceful lease handoff prevent duplicates?

No. Duplicates can still occur. The KCL migration page states that a graceful lease handoff significantly reduces the possibility of duplicates, but does not eliminate it. It also specifies that downstream components should be designed to be idempotent.

Q4. Where is the KCL lease table?

In this article's reading, the lease table is in DynamoDB in the account that runs KCL. No page states this in one sentence; this article derived it from three statements: the billing sentence, the ARN in the IAM permissions page, and the configuration in the Cross-Account Blog.

Q5. Does upgrading an MSK cluster to Kafka 4.x switch to the new rebalance protocol?

Not by itself. On the server side, the new protocol is enabled by default. However, on the client side, it will continue to use the classic protocol unless group.protocol is set to consumer. The Kafka 4.2 consumer configuration page gives classic as the default of group.protocol.

Q6. With the classic protocol, does every consumer pause on every rebalance?

It depends on the assignment strategy. The re:Post article on consumer lag states that all assignments are revoked during a rebalance. The Next-Generation Protocol Blog makes the impact on consumers in the classic protocol depend on the assignment strategy, and describes CooperativeStickyAssignor as a strategy that supports incremental and cooperative rebalancing. In the consumer protocol, only the affected partitions move, while other consumers continue their operations.

Q7. Can share groups be used with MSK?

As of November 1, 2026, no statement was found in the sources of a configuration in which share groups can be used in production with MSK (Section 5.9). The MSK versions page describes Queues in Kafka 4.1 as a preview feature, states that Kafka 4.2 is available only on Express brokers, and states that KIP-932 is not yet supported on Express brokers. The Lambda Queue mode page states that MSK does not currently support Queue mode.

Q8. How does Lambda's Kafka ESM assign partitions?

The Lambda developer guide states that for self-managed Kafka in Stream mode, Lambda assigns each partition to a single consumer. No statement was found on whether this assignment is determined by Kafka's group coordinator or within Lambda itself (Section 6.2).

Q9. What happens if a Lambda ESM is given the group ID of existing consumers?

Lambda will only receive a portion of the topic's messages. Kafka distributes messages among consumers in the same group. If a valid group with the same ID already exists, Lambda will disregard the StartingPosition and begin reading from the group's committed offset.

Q10. Does the Lambda Kinesis ESM use KCL?

No statement was found either way (Section 6.1). The sources state that an ESM is a resource managed in the Lambda service, and that one batch holds records from only one shard.

12. Summary

The location where consumer assignments are determined varies depending on the type of consumer. In KCL, the user's worker determines the assignment, and the result is stored in the lease table. In this article's reading, the lease table is in the account that runs KCL. KCL 3.x defaults to balancing CPU utilization or throughput, but does not attempt to equalize the number of leases. Even with handoff, the possibility of duplicates remains.

In Kafka, the group coordinator manages participants. In the classic protocol, a client leader calculates assignments, while in the consumer protocol, the broker calculates them. Kafka 4.x enables the consumer protocol by default on the server side, but defaults to the classic protocol on the client side. According to the Next-Generation Protocol Blog, MSK makes the consumer protocol available across all versions 4.x on both Standard and Express brokers, while the MSK versions page lists Kafka 4.2 as available only on Express brokers; in this article's reading, the blog means the 4.x versions that each broker type offers. No statement was found, in the MSK sources as of November 1, 2026, of a configuration that uses share groups in production on MSK (Section 5.9).

In Lambda's ESM, the AWS-side event poller reads the data, but the documentation provides little information about how assignments are determined or where the state is stored. Only results such as IteratorAge and OffsetLag are visible.

Whether you add consumers, lose one, upgrade versions, or share group names, first check in which layer your consumer's assignment is decided, and then examine the results using the methods available for that layer.

13. References

Amazon Kinesis Data Streams Developer Guide: KCL concepts
Amazon Kinesis Data Streams Developer Guide: DynamoDB metadata tables and load balancing in KCL
Amazon Kinesis Data Streams Developer Guide: KCL configurations
Amazon Kinesis Data Streams Developer Guide: Migrate from KCL 2.x to KCL 3.x
Amazon Kinesis Data Streams Developer Guide: Roll back to the previous KCL version
Amazon Kinesis Data Streams Developer Guide: Single table format for KCL
Amazon Kinesis Data Streams Developer Guide: IAM permissions required for KCL consumer applications
Amazon Kinesis Data Streams Developer Guide: KCL version lifecycle policy
Amazon Kinesis Data Streams Developer Guide: KCL 1.x and 2.x information
Amazon Kinesis Data Streams Developer Guide: Document history
Amazon DynamoDB Developer Guide: Using the DynamoDB Streams Kinesis adapter to process stream records
Amazon DynamoDB Developer Guide: Migrating from KCL 1.x to KCL 3.5.x+
Amazon MSK Developer Guide: Supported Apache Kafka versions
Amazon MSK Developer Guide: Troubleshoot your Amazon MSK cluster
Amazon MSK Developer Guide: Express brokers read-only configurations
AWS Lambda Developer Guide: Customizable consumer group ID in Lambda
AWS Lambda Developer Guide: Apache Kafka event poller scaling modes in Lambda
AWS Lambda Developer Guide: Low latency processing for Kafka event sources
AWS Lambda Developer Guide: Polling Kafka topics in Queue mode
AWS Lambda Developer Guide: Using Lambda to process records from Amazon Kinesis Data Streams
AWS Lambda Developer Guide: Using AWS Lambda with Amazon DynamoDB
AWS Lambda Developer Guide: How Lambda processes records from stream and queue-based event sources
AWS Lambda Developer Guide: Types of metrics for Lambda Event Functions
AWS Lambda API Reference: CreateEventSourceMapping
AWS Lambda API Reference: SelfManagedKafkaEventSourceConfig
AWS Lambda API Reference: AmazonManagedKafkaEventSourceConfig
AWS Lambda API Reference: EventSourceMappingMetricsConfig
AWS Big Data Blog: Reduce your compute costs for stream processing applications with Kinesis Client Library 3.0
AWS Big Data Blog: Best practices for scaling large consumer groups on Amazon MSK
AWS Big Data Blog: Optimize consumer rebalancing on Amazon MSK with next generation protocol
AWS Big Data Blog: Build cross-account streaming pipelines with the Amazon Kinesis Producer Library and Kinesis Client Library
AWS Big Data Blog: Building serverless event streaming applications with Amazon MSK and AWS Lambda
AWS Architecture Blog: Building resilient real-time streaming workers with Amazon DynamoDB leases
AWS re:Post Knowledge Center: Troubleshoot continuous rebalancing of your Amazon MSK consumer group
AWS re:Post Knowledge Center: I experience high consumer lag in my cluster
AWS re:Post Knowledge Center: Troubleshoot a blocked or stuck KCL application for Kinesis Data Streams
Apache Kafka 4.2 Documentation: Consumer Rebalance Protocol
Apache Kafka 4.2 Documentation: Consumer Configs
Apache Kafka 4.2 Documentation: Broker Configs
Apache Kafka Blog: Apache Kafka 4.2.0 Release Announcement
GitHub: awslabs/amazon-kinesis-client CHANGELOG

References:
Tech Blog with curated related content

Written by Hidekazu Konishi