Where a Kinesis Consumer Resumes After a Shard Splits or Merges - Parent Shards, Child Shards, KCL Leases, and Lambda
First Published:
Last Updated:
Who handles these three things depends on the type of consumer. If you call
GetRecords yourself, you learn that the parent has ended from the response in which NextShardIterator is null and from the ChildShards in that response, and you decide where to read the child from. If you use the Kinesis Client Library (KCL), the KCL creates the leases for the child shards and, in the KCL 3.5.4 implementation, holds back the processing of a child, if the parent's lease exists, until the parent's checkpoint becomes SHARD_END. That checkpoint is written when the record processor checkpoints in shardEnded. Where the KCL starts the lease that it creates for a child at the end of the parent is also written only in the KCL source. For reading Kinesis through an AWS Lambda event source mapping (ESM), no statement on how parents and children are handled was found in the Lambda Developer Guide. The statement that Lambda processes the parent first is on the DynamoDB Streams page of the DynamoDB Developer Guide, and it is about the DynamoDB Streams ESM.For your own
GetRecords calls, enhanced fan-out, the KCL, the Lambda ESM, Amazon Managed Service for Apache Flink, Amazon Data Firehose, and DynamoDB Streams with the Kinesis Adapter, this article sets out in a table how each one learns that the parent has ended, where it starts in the child, and where the position is kept. It separates what the documentation states, what only the KCL source states, what only a blog post states, and what was not found anywhere.This article is not a resharding procedure, and it is not a guide to deciding when to add shards. It does not explain the ordering rules, and it does not cover pricing. The sources were checked on November 2, 2026.
Related articles on this site:
- Where a Stream Consumer Can Start on AWS - Starting Positions in DynamoDB Streams, Kinesis Data Streams, KCL, and Lambda Event Source Mappings
- What Breaks Ordering, Service by Service on AWS - SQS FIFO, SNS FIFO, Kinesis, Amazon MSK, DynamoDB Streams, and the EventBridge Custom Event Bus
- AWS Real-Time Streaming Data Pipeline Architecture Guide - Ingestion, Processing, and Delivery with Kinesis, Managed Service for Apache Flink, and OpenSearch
- AWS History and Timeline regarding Amazon Kinesis - Overview, Functions, Features, Summary of Updates, and Introduction
- AWS Messaging and Event Routing Decision Guide - Choosing Between SQS, SNS, EventBridge, and Kinesis
Table of Contents
- 1. Scope and Assumptions
- 2. How to Read the Tables
- 3. Parent Shards and Child Shards
- 4. How a Consumer Learns That the Parent Shard Has Ended
- 5. Where to Begin Reading the Child Shard
- 6. The Position by Type of Consumer
- 7. On-Demand Mode and Service-Managed Record Distribution
- 8. What This Setup Showed
- 9. Where the Sources Disagree and What They Do Not State
- 10. How It Goes Wrong
- 11. Frequently Asked Questions About Kinesis Resharding and Consumer Positions
- 12. Summary
- 13. References
1. Scope and Assumptions
This section first sets out the intended readers, the scope, the sources read, and how this article tells apart terms that share a spelling. The conclusions, stated first, are these eight:- A split or a merge puts the parent shard in the
CLOSEDstate, and the parent moves to theEXPIREDstate after the stream's retention period has passed. While it isCLOSED, the records in the parent shard can be read. A split cannot produce more than two shards and a merge cannot take more than two shards in one operation, andSplitShardandMergeShardscan be used only on provisioned streams.
- With your own
GetRecordscalls, you learn that the parent has ended from the response in whichNextShardIteratoris null.ChildShardsis returned only in the response that reaches the end of the shard. With enhanced fan-out, you learn it from the event in whichContinuationSequenceNumberis null and from theChildShardsin the event at the end of the shard.
- On where to read a child shard from, the documentation states two things: if the order of the data is important, read the parent shards to the end before reading the child shard, and if the child came from a merge, read the other parent shard to the end too. Both statements carry the condition that order is important. No statement was found on which position type to start a child shard with (Section 5.2).
- The KCL chapter of the developer guide states that the data that was in the shards before the resharding is processed first. In the KCL 3.5.4 source, the lease that the KCL creates for a child shard at the end of the parent has a checkpoint of
TRIM_HORIZON, and if a lease for the parent shard exists, the child is not processed until the parent's checkpoint becomesSHARD_END. These are implementation facts, not documented promises (Section 5.3).
- Per the KCL Java consumer page and the Javadoc in the KCL 3.5.4 source, a KCL record processor has to write a checkpoint inside
shardEnded, and the Javadoc states that if it does not, the child shards make no progress.
- For the Lambda Kinesis ESM, no statement that it processes the parent shard first was found in the Lambda Developer Guide (Section 6.4). The statement that the parent shard is processed first is on the DynamoDB Streams page of the DynamoDB Developer Guide, and it is about the DynamoDB Streams ESM.
- The Flink connector blog post states that the new Kinesis source connector for Managed Service for Apache Flink follows the parent-child lineage and that the legacy
FlinkKinesisConsumersource could not. No statement was found on how Firehose handles resharding (Section 6.6).
- The developer guide states that an on-demand stream splits a shard within 15 minutes when the incoming traffic to that shard exceeds 500 KB/s. No statement that an on-demand stream merges shards was found in the developer guide (Section 7.2); it appears only in one sentence of the warm throughput blog post. Service-managed record distribution, which the API sets with the value
AUTO, does not change the shard of records that are already in the stream.
1.1 Intended Audience
This article is for developers and operators who change the number of shards in Kinesis Data Streams by hand, withUpdateShardCount, or by leaving it to on-demand mode. In particular, it is for readers who believe any of the following:- Consumers automatically resume reading from where they left off after a shard is split.
- The sequence number of a parent shard can be used as is on its child shards.
- Lambda, like the KCL, reads the parent shard first.
- If you leave it to on-demand mode, you do not need to think about splits or merges.
For each of these, what the documentation covers differs by type of consumer. They fall into three groups: ones the documentation says hold, ones it says do not hold, and ones for which no statement either way was found.
1.2 Scope and Exclusions
This article covers the position across a shard boundary. Specifically, it covers the following:- The states and lineage of parent and child shards, and the constraints on splits and merges.
- How a consumer learns that a parent shard has ended.
- Where to start reading a child shard, and the two parent shards of a child created by a merge.
- How your own
GetRecordscalls, enhanced fan-out, each KCL version, the Lambda Kinesis and DynamoDB Streams ESMs, the new and legacy connectors of Managed Service for Apache Flink, Firehose, and DynamoDB Streams with the Kinesis Adapter handle it. - Automatic splitting in on-demand mode, and service-managed record distribution.
- Places where the sources disagree, and places where no statement was found in the sources.
This article does not cover the following:
- Pricing and billing mechanisms.
- The ordering guarantees of each service. This site's What Breaks Ordering, Service by Service on AWS covers them. This article treats the rule of reading the parents to the end before the child only as a matter of position.
- The list of values for where a consumer can start in a stream. This site's Where a Stream Consumer Can Start on AWS covers it. That article leaves shard splits and merges out of scope, and this article covers that part.
- Deciding when and how much to reshard, and the procedure. This site's AWS Real-Time Streaming Data Pipeline Architecture Guide covers it.
- How it is decided which consumer owns which shard. Another article in this series, Who Assigns Shards and Partitions to a Stream Consumer on AWS, covers it. This article covers only the part where the KCL creates the leases of the child shards and waits for the parent to end.
- Where a batch that the function failed goes, and where reading resumes within a single shard. Another article in this series, What Happens to a Failed Batch in a Lambda Event Source Mapping, covers it.
- Kinesis Video Streams, Apache Kafka, and Amazon MSK.
1.3 Date of Review and Materials Reviewed
This article read the following sources on November 2, 2026. The text uses the short names in parentheses.- The pages for
GetRecords,GetShardIterator,SubscribeToShard,SubscribeToShardEvent,ChildShard,Shard,ListShards,ShardFilter,SplitShard,MergeShards,UpdateShardCount,PutRecord, andUpdateStreamRecordDistributionStrategyin the Amazon Kinesis Data Streams API Reference. The text calls each of these the API name followed by "page", such as theGetRecordspage. - In the Amazon Kinesis Data Streams Developer Guide: Reshard a stream (the resharding page), Complete the resharding action (the resharding completion page), Merge two shards (the merge page), Develop shared-throughput consumers with the AWS SDK for Java (the shared-throughput consumer page), Develop enhanced fan-out consumers with the AWS SDK for Java (the enhanced fan-out consumer page), Choose the right mode to stream in (the capacity mode page), How service-managed record distribution works (the service-managed page), Consumer considerations in the service-managed section (the service-managed consumer page), and Amazon Data Firehose consuming from service-managed streams (the Firehose integration page).
- In the KCL chapter of the same developer guide: Use resharding, scaling, and parallel processing to change the number of shards (the KCL resharding page), DynamoDB metadata tables and load balancing in KCL (the KCL tables page), KCL configurations (the KCL configuration page), Develop consumers with KCL in Java (the KCL Java consumer page), KCL 1.x and 2.x information (the KCL 1.x and 2.x page), Monitor the Kinesis Client Library with Amazon CloudWatch (the KCL monitoring page), and KCL version lifecycle policy (the KCL lifecycle page).
- In the AWS Lambda Developer Guide: Using Lambda to process records from Amazon Kinesis Data Streams (the Lambda Kinesis page) and Implementing stateful Kinesis Data Streams processing in Lambda (the tumbling window page). To look for statements on parent and child shards, this article also opened the pages under the Lambda Kinesis page (creating the ESM, parameters, partial batch responses, on-failure destinations, tutorial, and filtering), the Lambda DynamoDB Streams page and the pages under it, the ESM page, the scaling page, the ESM logging page, and the best practices page.
- In the Amazon DynamoDB Developer Guide: Change data capture for DynamoDB Streams (the DynamoDB Streams page) and the DynamoDB Streams Kinesis Adapter page (the Adapter page). To look for the statement on about four hours, this article also opened the page on best practices with Lambda.
- In the Amazon Data Firehose Developer Guide: Configure source settings for Amazon Kinesis Data Streams (the Firehose source page).
- Two AWS re:Post articles: Decrease your Lambda function's IteratorAge metric (the IteratorAge re:Post article) and Change the number of open shards in Kinesis Data Streams (the open shards re:Post article).
- Four AWS blog posts: Introducing the new Amazon Kinesis source connector for Apache Flink on the Big Data Blog (the Flink connector blog post), Scale down Kinesis Data Streams on-demand capacity with ODA warm throughput on the Big Data Blog (the warm throughput blog post), Build scalable, event-driven architectures with Amazon DynamoDB and AWS Lambda on the Database Blog (the DynamoDB and Lambda blog post), and Process Large DynamoDB Streams Using Multiple Amazon Kinesis Client Library (KCL) Workers on the Big Data Blog (the DynamoDB Streams and KCL blog post).
- Two What's New posts: Amazon Kinesis Data Streams now supports scaling down ingest capacity with warm throughput (July 24, 2026) and Amazon Kinesis Data Streams announces Service-Managed Partition Keys for simplified data ingestion (September 23, 2026).
- In the KCL GitHub repository: the source, README, and CHANGELOG at tag v3.5.4, and the CHANGELOG on branch v1.x. The source code was obtained from raw.githubusercontent.com on November 2, 2026.
As of the verification date, the latest KCL tag was v3.5.4 (October 30, 2026).
1.4 Different Things with the Same Spelling
This article tells similar terms apart as follows:- Parent shard and child shard: In this article, parent and child shards are the parents and children that splits and merges create in Kinesis Data Streams. DynamoDB Streams shards also have parents and children, but the service creates those shards and splits them automatically. Section 6.7 covers DynamoDB Streams shards.
- checkpoint: In this article, checkpoint means the
checkpointvalue that the KCL keeps in each row of the lease table. It is kept apart from the position that the Lambda ESM keeps internally, the position that goes into a Managed Service for Apache Flink snapshot, and the partial batch response checkpoint that the failed-batch article in this series covers. The partial batch response checkpoint is about where to resume within one shard; this article is about how the position moves from a parent shard to a child shard. - Shard assignment: In this article, this means which consumer reads which shard, on the consumer side. Service-managed record distribution, from September 23, 2026, is a feature in which the service decides which shard a record written by a producer goes into, and it is a different layer.
- The two 5-minute limits: A shard iterator that
GetShardIteratorreturns expires 5 minutes after it is returned. A subscription made withSubscribeToShardends after at most 5 minutes. The former is aboutGetRecordsreaders, and the latter is about enhanced fan-out readers. - consumer: This article calls your own readers, KCL workers, Lambda ESMs, and similar readers consumers. The consumer that you register for enhanced fan-out is a registration for receiving read throughput.
2. How to Read the Tables
This section defines the table columns used in Chapters 3 through 9 and the terms for the state of the sources. The last two columns and the state terms are shared with the other articles in this series.2.1 Columns
The table in this article puts aConsumer column first, followed by these columns:How It Learns the Parent Ended: What tells that consumer it has read the parent shard to the end.Where It Starts in the Child: Where in the child shard it starts reading, and who decides that.Where the State Lives: Where the position it has read up to is kept.Where It Is Written: Where the statements in that row come from, by source and page.No Statement Found: Matters for which no statement was found in the sources in Section 1.3, even after changing the search terms and the kind of source. A cell with nothing in it shows—.
Where the State Lives is the same column name as in the assignment article of this series. A non-empty No Statement Found cell does not mean that the thing is absent or does not happen.2.2 Six States
The state of the sources is described with the following six terms. The first five are shared with the other articles in this series.- No statement found: In the text, this is written as
no statement was found, accompanied by the scope of the search. - Documentation does not list: The item is not present in a list or table. In the text, this is written as
the documentation does not list. - Sources disagree: In the text, this is written as
the sources disagree, and both statements, along with their sources, are presented. This article does not decide that either one is wrong. - Inference: In the text, the same sentence states that it is this article's reading and not a statement in any source.
- Implementation fact: What an implementation such as the KCL source states, which is not a documented promise. In the text, the same sentence says that it is an implementation fact and gives the version and the retrieval date.
- Observation: What was observed in the setup of Chapter 8. In the text, the same sentence names the Chapter 8 section whose setup it was. An observation does not override a statement in the sources.
2.3 Terminology
This article uses the following terms consistently:- parent shard: The original shard that a split or a merge closed.
- adjacent parent shard: The other parent shard of a child created by a merge. The API puts it in
AdjacentParentShardId. - child shard: A new shard that a split or a merge creates.
- worker: A KCL worker. It is the unit of KCL processing inside the process that you run.
- record processor: Your processing code, which the KCL worker creates for each shard it holds a lease for.
- lease: What the KCL keeps as one row of the lease table to represent the tie between a shard and a worker.
- event poller: The AWS-side resource inside a Lambda ESM that reads the stream.
- enhanced fan-out: A way of reading in which a registered consumer receives records from
SubscribeToShardwith dedicated throughput for each shard.
2.4 Strength of Sources
This article treats the sources as stronger in this order:- Statements in the developer guides and the API Reference.
- Statements in the README and CHANGELOG of the KCL project. They come from the developers of the KCL but are not the AWS developer guide, so the same sentence names the source.
- Statements in AWS blog posts and in the re:Post Knowledge Center. The same sentence names the blog post.
- The KCL source, written as implementation facts in the sense of Section 2.2.
3. Parent Shards and Child Shards
This section sets out what splits and merges do to parent and child shards, and the constraints on the operations, from the resharding completion page, the resharding page, the shared-throughput consumer page, and the API Reference.3.1 OPEN, CLOSED, EXPIRED
The resharding completion page describes three states of a parent shard.In the process of resharding, a parent shard transitions from an OPEN state to a CLOSED state to an EXPIRED state.
OPEN: Before a reshard operation, a parent shard is in the OPEN state, which means that data records can be both
added to the shard and retrieved from the shard.
CLOSED: After a reshard operation, the parent shard transitions to a CLOSED state. This means that data records are
no longer added to the shard. Data records that would have been added to this shard are now added to a child shard
instead. However, data records can still be retrieved from the shard for a limited time.
The same page also states that the records in the parent shard do not disappear in the resharding.
However, any data records that were in the parent shards before the reshard remain in those shards. The parent
shards do not disappear when the reshard occurs. They persist along with the data they contained before the reshard.
The same page states that the parent shard becomes
EXPIRED when the stream's retention period has passed and all the records in the parent shard have expired. The ListShards page states that expired shards are not included in the list.This action does not list expired shards.
In this article's reading, the unread records of a parent shard remain for the retention period even after the parent closes. This period is what lets a consumer read the parent shard to the end before it moves on to the child shard.
3.2 The Lineage of Splits and Merges
TheShard page shows the lineage in two fields. ParentShardId is the ID of the parent shard, and AdjacentParentShardId is the ID of the shard adjacent to the parent shard. The shared-throughput consumer page describes how these two fields are filled in a split and in a merge.In the case of a split, the two new shards both have parentShardId equal to the shard ID of the shard that you were
processing previously. The value of adjacentParentShardId for both of these shards is null.
In the case of a merge, the single new shard created by the merge has parentShardId equal to shard ID of one of the
parent shards and adjacentParentShardId equal to the shard ID of the other parent shard.
The
ChildShard page describes the shape of each element of ChildShards in a GetRecords response. ParentShards is an array of strings, and the ChildShard page describes its contents as the current shard that is the parent of the existing child shard. In this article's reading, the element for a child created by a merge can hold the IDs of two parent shards. In the setup of Section 8.2, the ParentShards of the element for the child created by the merge held the IDs of both parent shards.
3.3 Constraints on the Operations
The resharding page states that a split cannot produce more than two shards and a merge cannot take more than two shards in one operation.Resharding is always pairwise in the sense that you cannot split into more than two shards in a single operation,
and you cannot merge more than two shards in a single operation.
The
SplitShard, MergeShards, and UpdateShardCount pages state that each of these APIs can be used only on provisioned streams. The three pages have the same sentence.This API is only supported for the data streams with the provisioned capacity mode.
After an operation, you wait until the stream is back in the
ACTIVE state before the next one. The resharding completion page states you must wait for the stream to become active again., and the SplitShard page lists ResourceInUseException as the error returned when the resource is not in the ACTIVE state. For a statement that limits resharding to one operation at a time, no statement was found. The search covered the SplitShard, MergeShards, and UpdateShardCount pages, the resharding page, the resharding completion page, and the open shards re:Post article.UpdateShardCount is the API in which you specify a target number of shards and the service performs the splits and merges. The UpdateShardCount page states that short-lived shards can be created in addition to the final shards.To update the shard count, Kinesis Data Streams performs splits or merges on individual shards. This can cause
short-lived shards to be created, in addition to the final shards.
In this article's reading, comparing only the number of shards before and after
UpdateShardCount does not tell you how many splits and merges took place. A consumer also has to read the short-lived shards to the end as parent shards.3.4 How to Check
You check the state and lineage of shards withListShards. The merge page states that an open shard is the one whose ending sequence number is null.Shards that are OPEN—that is, not CLOSED—have an ending sequence number of null.
The
ShardFilter of ListShards selects the shards to include in the list by type. The ShardFilter page lists six types: AFTER_SHARD_ID, AT_TRIM_HORIZON, FROM_TRIM_HORIZON, AT_LATEST, AT_TIMESTAMP, and FROM_TIMESTAMP. The default setting is FROM_TRIM_HORIZON, which returns every shard in the retention period. AT_LATEST returns only the currently open shards. The descriptions of FROM_TIMESTAMP on the ShardFilter page and the ListShards page disagree. Section 9.1 puts both statements side by side.4. How a Consumer Learns That the Parent Shard Has Ended
This section compares, for each way of reading, what tells a consumer that it has read the parent shard to the end.4.1 GetRecords — NextShardIterator and ChildShards
TheGetRecords page describes the response when a closed shard has been read to the end.If the shard has been closed, the shard iterator can't return more data and GetRecords returns null in
NextShardIterator.
The same page limits when
ChildShards is returned.ChildShards
The list of the current shard's child shards, returned in the GetRecords API's response only when the end of the
current shard is reached.
The Adapt to a reshard section of the shared-throughput consumer page puts these two together and describes how to learn that the parent has ended and how to find the child shards.
If getRecordsResult.getNextShardIterator returns null, it indicates that a shard split or merge has occurred that
involved this shard. This shard is now in a CLOSED state and you have read all available data records from this
shard.
In this scenario, you can use getRecordsResult.childShards to learn about the new child shards of the shard that is
being processed that were created by the split or merge.
In this article's reading, the signal that tells your own
GetRecords reader that the parent has ended is the response in which NextShardIterator is null. ChildShards is returned only in that response, so if you discard it, you have to look up the child shard IDs again with ListShards.4.2 The Iterator of a Closed Parent Shard and the 5-Minute Iterator Lifetime
GetShardIterator returns an iterator for a closed shard too. The GetShardIterator page states:If the shard is closed, GetShardIterator returns a valid iterator for the last sequence number of the shard.
A shard iterator has a lifetime. The
GetShardIterator page states A shard iterator expires 5 minutes after it is returned to the requester., and the shared-throughput consumer page states that it stays valid for 5 minutes even after it is used.A shard iterator is valid for 5 minutes. If you use a shard iterator while it is valid, you get a new one. Each
shard iterator remains valid for 5 minutes, even after it is used.
If you use an expired iterator,
GetRecords returns ExpiredIteratorException. These 5 minutes apply to GetRecords readers and are not the 5 minutes of the subscription in Section 4.3. In this article's reading, when the reader of a parent shard has stopped for more than 5 minutes, it can continue the parent shard from where it left off by getting a new iterator with AFTER_SEQUENCE_NUMBER from the last sequence number it processed. This site's Where a Stream Consumer Can Start on AWS covers the iterator lifetime in detail.4.3 Enhanced Fan-Out — ContinuationSequenceNumber and ChildShards
With enhanced fan-out, when you subscribe withSubscribeToShard, SubscribeToShardEvent events arrive. The SubscribeToShardEvent page describes ChildShards as follows.ChildShards
The list of the child shards of the current shard, returned only at the end of the current shard.
The enhanced fan-out consumer page describes the signal that the parent has ended in terms of
ContinuationSequenceNumber.If event.ContinuationSequenceNumber returns null, it indicates that a shard split or merge has occurred that involved
this shard. This shard is now in a CLOSED state, and you have read all available data records from this shard.
ContinuationSequenceNumber is also the value that an enhanced fan-out reader keeps as its position. The SubscribeToShardEvent page states:Use ContinuationSequenceNumber for checkpointing because it captures your shard progress even when no data is
written to the shard.
A subscription has a maximum length. The
SubscribeToShard page states that a subscription lasts up to 5 minutes and that you call SubscribeToShard again to keep receiving records.for up to 5 minutes, after which time you need to call SubscribeToShard again to renew the subscription if you want
to continue to receive records.
For a statement that the subscription closes when it reaches the end of the shard,
no statement was found. The search covered the SubscribeToShard page, the SubscribeToShardEvent page, and the enhanced fan-out consumer page. In the setup of Section 8.2, the subscription ended 1 to 5 milliseconds after the end event, but this is not a documented promise.4.4 KCL — shardEnded and SHARD_END
With the KCL, it is the KCL that tells the record processor that the parent shard has ended. As an implementation fact, the Javadoc ofshardEnded in ShardRecordProcessor in the KCL 3.5.4 source, retrieved on November 2, 2026, states the following. The the all in it is as in the original.Called when the shard that this record process is handling has been completed. Once a shard has been completed no
further records will ever arrive on that shard.
When this is called the record processor <b>must</b> call {@link RecordProcessorCheckpointer#checkpoint()},
otherwise an exception will be thrown and the all child shards of this shard will not make progress.
In the implementation, calling
checkpoint() inside shardEnded sets the checkpoint of that shard's lease to SHARD_END, and the ShutdownTask in the same source throws an exception that contains Application must checkpoint upon shard end. when the checkpoint is not SHARD_END. The KCL Java consumer page of the developer guide states, for shardEnded, Checkpointing in this method is mandatory to complete processing. The KCL resharding page, the KCL tables page, the KCL configuration page, and the KCL 1.x and 2.x page do not state it.How the KCL learns that a shard has ended changed between versions. The KCL 1.x and 2.x page states that the KCL uses
ChildShards from KCL 2.3 on, and from KCL 1.14 on in 1.x.KCL 2.3 uses the ChildShards return parameter of the GetRecords and the SubscribeToShard APIs to perform lease/shard
synchronization that happens at SHARD_END for closed shards, allowing a KCL worker to only create leases for the
child shards of the shard it finished processing.
Section 6.3 covers the difference before and after this change.
5. Where to Begin Reading the Child Shard
For where reading of a child shard starts, this section separates what the documentation states, what it does not state, and what the KCL implementation does.5.1 What the Documentation States — If Order Matters, Read the Parents to the End First
The resharding completion page states that you can read the child shards right away, and gives the reason not to.After the reshard has occurred and the stream is again in an ACTIVE state, you could immediately begin to read data
from the child shards. However, the parent shards that remain after the reshard might still contain data that you
haven't read yet that was added to the stream before the reshard. If you read data from the child shards before
having read all data from the parent shards, you could read data for a particular hash key out of the order given by
the data records' sequence numbers. Therefore, assuming that the order of the data is important, you should, after a
reshard, always continue to read data from the parent shards until it is exhausted. Only then should you begin
reading data from the child shards.
This statement carries the condition
assuming that the order of the data is important. If order is not important, the page does not stop you from starting to read the child shards right away.A child shard created by a merge has two parent shards. The shared-throughput consumer page writes about the other parent shard, the one besides the parent for which
NextShardIterator returned null.Your application has already read all the data from one of these shards. This is the shard for which
getRecordsResult.getNextShardIterator returned null. If the order of the data is important to your application,
ensure that it also reads all the data from the other parent shard before reading any new data from the child shard
created by the merge.
This statement also carries the condition
If the order of the data is important to your application. In this article's reading, what to check before reading a child created by a merge is whether you have read to the end every parent shard ID that the ChildShards element holds. If you move on to the child after the end of only one parent, you read the new records of the child before the unread records of the other parent.The resharding completion page also states that records that would have gone into the parent shard after it closed go into a child shard instead (Section 3.1). In this article's reading, after the parent shard has ended, there is no need to wait for records to arrive late in the parent shard.
The ordering rules themselves are covered, together with the KCL statements on ordering, in this site's What Breaks Ordering, Service by Service on AWS.
5.2 What the Documentation Does Not State — The Position Type and Sequence Numbers in the Child Shard
For a statement on which position type, such asTRIM_HORIZON, AT_SEQUENCE_NUMBER, or AT_TIMESTAMP, to start reading a child shard with, no statement was found. The search covered the GetShardIterator page, the resharding completion page, the shared-throughput consumer page, the enhanced fan-out consumer page, the KCL resharding page, the KCL 1.x and 2.x page, and the KCL README.For a statement that the sequence number of a parent shard cannot be used with
AT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER on a child shard, no statement was found either. The closest statement about the range of sequence numbers is this one on the PutRecord page.Sequence numbers increase over time and are specific to a shard within a stream, not across all shards within a
stream.
The KCL 1.x and 2.x page describes the lease
checkpoint as This value is unique across all shards in the data stream. A value being unique in the stream and values being comparable across shards are different properties, so this article does not read these two statements as a disagreement. In the setup of Section 8.3, when the last sequence number of the parent shard was passed as AFTER_SEQUENCE_NUMBER for the child shard, GetShardIterator returned InvalidArgumentException.In this article's reading, a child shard starts empty at the time of the resharding and receives only the records that come after it. The
GetShardIterator page describes TRIM_HORIZON as the position that reads from the oldest record in the shard that has not expired. So if a child shard is younger than the retention period, reading it with TRIM_HORIZON reads it from its first record. This is this article's reading derived from two statements, not a statement that says to read child shards with TRIM_HORIZON.5.3 The KCL Implementation — The Child Lease Starts at TRIM_HORIZON and Waits for the Parent's SHARD_END
The KCL chapter of the developer guide states that child shards are processed after their parent shards. The KCL tables page describesparentShardId and childShardId in the KCL 3.x lease table as follows.parentShardId: ID of this shard's parent. Makes sure that the parent shard is fully processed before processing
starts on the child shards, maintaining the correct record processing order.
childShardId: List of child shard IDs resulting from this shard's split or merge. Used to track shard lineage and
manage processing order during resharding operations.
The KCL monitoring page names the step that waits for the parent shard as the
BlockOnParentTask operation.The BlockOnParentTask operation ensures that record processing for the new shards does not start until the parent
shards are completely processed by the KCL.
The README of the KCL project at tag v3.5.4 states the same thing.
It maintains ordering by making sure that child shards are processed only after their parent shard is completed and
checkpointed.
Which position the lease of a child shard starts from is written only in the source. As an implementation fact, the
HierarchicalShardSyncer in the KCL 3.5.4 source, retrieved on November 2, 2026, creates the lease of a child shard as follows.Generate a lease object for the given Child Shard. Checkpoint is set to TRIM_HORIZON
...
newLease.checkpoint(ExtendedSequenceNumber.TRIM_HORIZON);
In the implementation, waiting for the parent shard is also conditional. The class comment of
BlockOnParentShardTask in the same source states:We check if we have checkpoint(s) for the parent shard(s).
If a checkpoint for a parent shard is found, we poll and wait until the checkpoint value is SHARD_END
(application has checkpointed after processing all records in the shard).
If we don't find a checkpoint for the parent shard(s), we assume they have been trimmed and directly
proceed with processing data from the shard.
In the implementation, if the lease of the parent shard is in the lease table, the child shard is not processed until the parent's
checkpoint becomes SHARD_END, and if there is no lease for the parent shard, the child goes ahead without waiting. The interval at which it checks the parent shard while waiting is parentShardPollIntervalMillis on the KCL configuration page, and the default is 10,000 milliseconds (10 seconds).In the implementation, the lease of a child created by a merge is created when the leases of the two parent shards are both in the table or both absent. The
ShutdownTask in the same source states:// For child shard resulted from merge of two parent shards, verify if both the parents are either present or
// not present in the lease table before creating the lease entry.
When only one of the two parent leases is in the table, the same source postpones creating the child's lease with
Hence deferring lease creation of child shard. In this article's reading, with the KCL, processing of a child created by a merge starts after the child's lease has been created and, in addition, after every parent lease still in the table has a checkpoint of SHARD_END. This is this article's reading derived from three statements in the source, not a procedure that the KCL chapter of the developer guide promises.6. The Position by Type of Consumer
This section gathers Chapters 4 and 5 by type of consumer and lines them up in the table in Section 6.8.6.1 Your Own GetRecords Calls
You learn that the parent has ended from the response in whichNextShardIterator is null. You learn the IDs of the child shards from the ChildShards of that response or from ListShards. You decide where to read the child shard from. If order is important, read all the parent shards of a child to the end before reading the child, as the two statements in Section 5.1 say.You also decide where to keep the position. For a statement on where to keep it,
no statement was found; the search covered the GetRecords page, the shared-throughput consumer page, and the resharding completion page. Because an iterator expires in 5 minutes, when you resume from a kept position, you get a new iterator with AFTER_SEQUENCE_NUMBER from the kept sequence number.6.2 Readers That Use Enhanced Fan-Out Directly Through the API
You learn that the parent has ended from the event in whichContinuationSequenceNumber is null and from the ChildShards in the event at the end of the shard. A subscription ends after at most 5 minutes, so when a subscription ends partway through a parent shard, you call it again from the last ContinuationSequenceNumber. The SubscribeToShardEvent page states that ContinuationSequenceNumber is used as the SequenceNumber for AT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER in the next SubscribeToShard.For a statement on which position to start subscribing to a child shard from,
no statement was found; the search covered the SubscribeToShard page, the SubscribeToShardEvent page, and the enhanced fan-out consumer page.6.3 KCL — Differences by Version
The KCL resharding page states that the KCL does not start resharding, that it adjusts to resharding, and that it processes the data from before the resharding first.When new shards are created as a result of resharding, the KCL discovers the new shards and populates new rows in
the table.
...
The KCL ensures that any data that existed in shards prior to the resharding is processed first.
The position is kept in the
checkpoint of the row for each shard in the DynamoDB lease table that the KCL creates. Which account the lease table is in, and the assignment, are covered by the assignment article in this series, Who Assigns Shards and Partitions to a Stream Consumer on AWS. The initial position in the KCL configuration is used only when a lease is created; this site's Where a Stream Consumer Can Start on AWS covers this point. The lease of a child shard created at the end of the parent is, in the implementation, created with TRIM_HORIZON, as Section 5.3 shows. The code in HierarchicalShardSyncer that creates these leases at the end of the parent does not refer to the initial position setting. In the same source, the shard sync gives the checkpoint of the initial position to a lease that it creates for a shard with no ancestor lease in the table, including every shard when the table is empty. In the setup of Section 8.4, the shard sync created the leases of the two shards that existed at the start with the initial position, which was TRIM_HORIZON, and the ShutdownTask created the leases of the child shards; because both gave TRIM_HORIZON, the setup does not tell the two paths apart.How the KCL finds child shards changed between versions. The KCL 1.x and 2.x page states that from KCL 1.14 and KCL 2.3, a worker learns only about the children of the shards it processed.
With the above changes, the behavior of KCL is moving from the model of all workers learning about all existing
shards to the model of workers learning only about the children shards of the shards that each worker owns.
Therefore, in addition to the synchronization that happens during consumer application bootstraping and reshard
events, KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the
lease table (in other words, to learn about all new shards) to ensure the complete hash range of the data stream is
being processed and create leases for them if required.
The same page states that the component in charge of this periodic scan is
PeriodicShardSyncManager. It also states that from KCL 1.14, LeaseCleanupManager deletes leases that have reached SHARD_END asynchronously. The KCL configuration page gives TRUE as the default of cleanupLeasesUponShardCompletion and states When set, leases are removed as soon as the child leases have started processing.The KCL configuration page gives
FALSE as the default of ignoreUnexpectedChildShards and states:When set, child shards that have an open shard are ignored. This is primarily for DynamoDB Streams.
The 1.8.9 entry of the CHANGELOG on the KCL v1.x branch gives the reason for adding this setting as a race in which a parent shard appears open while it has child shards.
There is a race condition where it's possible for the a parent shard to appear open, while having child shards.
The KCL 3.x CHANGELOG states
Shard lifecycle management is improved to deal with edge cases around shard splits and merges to ensure records continue being processed as expected. in the 3.0.0 entry (November 6, 2024), and Fix FanoutRecordsPublisher restart behavior at SHARD_END in the 3.5.0 entry (June 22, 2026). In this article's reading, the KCL's handling of the end of a shard has been fixed over several versions, so it is worth pinning the version when you check it.For the phase of KCL 1.x, this article copies the two columns of the table on the KCL lifecycle page as they are. In that row, Current phase is
Maintenance mode and End-of-support date is 2026-01-30.6.4 The Lambda ESM — Kinesis and DynamoDB Streams
The Lambda Kinesis page describes the order within a shard as follows.Lambda processes records in each shard in order. It stops processing additional records in a shard if your function
returns an error.
For a statement that the Lambda Kinesis ESM processes parent shards before child shards,
no statement was found. The search covered the Lambda Kinesis page and the seven pages under it, the ESM page, the scaling page, the ESM logging page, the best practices page, and the IteratorAge re:Post article. No statement on parent and child shards was found on the Lambda DynamoDB Streams page and the pages under it either. Under the Lambda Kinesis page and the Lambda DynamoDB Streams page, the only pages that touch parent and child shards are their tumbling window pages.For the Lambda Kinesis ESM, the only statement that touches child shards was this one on the tumbling window page.
Tumbling window aggregations do not support resharding. When a shard ends, Lambda considers the current window to be
closed, and any child shards start their own window in a fresh state.
The IteratorAge re:Post article describes
IteratorAge after a split as follows.Shard splitting doesn't have an immediate effect on a function's iterator age. Existing records remain in the shards
where the stream originally wrote them. Those shards must catch up on their backlog before the iterator age for the
shards decreases.
For where the Lambda ESM keeps the position it has read up to,
no statement was found. The assignment article in this series reaches the same conclusion.For the DynamoDB Streams ESM, the DynamoDB Streams page states:
When you use AWS Lambda to consume a stream through an event source mapping, Lambda processes each open shard with a
single function instance and reads that shard's records in sequence-number order. Lambda also follows shard lineage
for you, processing a parent shard before its child shards.
This statement is in the DynamoDB Developer Guide and is about the DynamoDB Streams ESM. This article does not extend it to the Kinesis ESM. In the setup of Section 8.5, the Lambda Kinesis ESM processed the first batch of each child shard after the last batch of its parent shard, but only a few unprocessed records were left in the parent shard, and this observation is not a documented promise.
6.5 Managed Service for Apache Flink — New and Legacy Connectors
The Flink connector blog post describes the new Kinesis source connector in version 5.0.0 of the AWS connectors, which the Apache Flink community released on November 11, 2024. The same post states that the legacy source could not follow the lineage.One known limitation of the legacy Kinesis source is that it was unable to follow the parent-child shard lineage.
As a consequence, ordering could not be guaranteed when resharding happens.
About the new connector, the same post states:
With the new connector, ordering is retained also when resharding happens. This is achieved following the
parent-child shard lineage, and consuming all records from a parent shard before proceeding with the child shard.
The same post states that
UniformShardAssigner, the default assigner of the new connector, assigns shards by looking at the HashKeyRange of each shard, and that the legacy FlinkKinesisConsumer spread shards, not partition keys, evenly across the parallel subtasks. The same post states that Flink sources typically save the position in checkpoints and savepoints, which Managed Service for Apache Flink calls snapshots. For a statement on which position in the child shard to read from, no statement was found. This post is the only source this article read about Flink.6.6 Amazon Data Firehose
The Firehose source page describes how Firehose reads when Kinesis Data Streams is the source.Amazon Data Firehose starts reading data from the LATEST position of your Kinesis stream.
...
Amazon Data Firehose calls the Kinesis Data Streams GetRecords operation once per second for each shard.
For streams that use service-managed record distribution, the Firehose integration page states
Firehose reads records from shards and delivers them to destinations regardless of how records were distributed across shards. For a statement on how Firehose handles resharding, or on whether it reads the parent shard or the child shard first, no statement was found. The search covered the Firehose source page, the Firehose integration page, and the three resharding pages of the Kinesis Data Streams Developer Guide.6.7 DynamoDB Streams and Kinesis Adapter
DynamoDB Streams shards also have a lineage. The DynamoDB Streams page states that an application has to process the parent shard first and that, with the Kinesis Adapter, the Adapter takes care of it.Because shards have a lineage (parent and children), an application must always process a parent shard before it
processes a child shard. This helps make sure that the stream records are also processed in the correct order.
(If you use the DynamoDB Streams Kinesis Adapter, this is handled for you.
The Adapter page includes finding the child shards of a closed parent shard in its list of the work that the Adapter's worker does.
Checks and enumerates child shards of a closed parent shard within the stream
For readers that do not use the Adapter, the DynamoDB Streams page describes finding the child shards by specifying the parent shard in the
ShardFilter of DescribeStream.By specifying a parent shard in the request, DynamoDB Streams will return its immediate child shards.
For a statement that DynamoDB Streams shards turn over about every four hours,
no statement was found in the DynamoDB Developer Guide. The search covered the DynamoDB Streams page, the Adapter page, and the page on best practices with Lambda. The figure is in two blog posts. The DynamoDB and Lambda blog post states A stream shard in DynamoDB has a lifespan of up to 4 hours., and the DynamoDB Streams and KCL blog post states close for writes roughly every four hours after they are created.The KCL configuration page describes the
ignoreUnexpectedChildShards setting from Section 6.3 as mainly for DynamoDB Streams.6.8 Table — Positioning by Consumer Type
| Consumer | How It Learns the Parent Ended | Where It Starts in the Child | Where the State Lives | Where It Is Written | No Statement Found |
|---|---|---|---|---|---|
Your own GetRecords | The response in which NextShardIterator is null. ChildShards is returned only in that response | You decide. If order is important, after reading all the parent shards to the end | You decide | GetRecords page, shared-throughput consumer page, resharding completion page | The position type in the child shard, where the position is kept |
| Reader that uses enhanced fan-out directly through the API | The event in which ContinuationSequenceNumber is null, and the ChildShards of the event at the end | You decide | You decide. ContinuationSequenceNumber is used as the position | SubscribeToShardEvent page, enhanced fan-out consumer page, SubscribeToShard page | The position type in the child shard, whether the subscription closes at the end |
| KCL 1.14 and later, 2.3 and later, and KCL 3.x | The GetRecords or SubscribeToShard response's ChildShards. The record processor gets a shardEnded call | The KCL creates the child shard leases. In the KCL 3.5.4 implementation, the leases created at the end of the parent start at TRIM_HORIZON, and if a parent shard lease exists, the child waits for SHARD_END | checkpoint in the DynamoDB lease table that the KCL creates | KCL resharding page, KCL tables page, KCL 1.x and 2.x page, KCL Java consumer page, KCL 3.5.4 source | A documented statement of the position type in the child shard |
| KCL 1.13 and earlier, and 2.2 and earlier | All workers sync with the list of shards | The KCL creates the child shard leases | checkpoint in the lease table | KCL 1.x and 2.x page | The position type in the child shard |
| Lambda Kinesis ESM | no statement was found | no statement was found. A tumbling window starts fresh in the child shard | no statement was found | Lambda Kinesis page, tumbling window page, IteratorAge re:Post article | Whether it processes the parent shard first, the position in the child shard, where the position is kept |
| Lambda DynamoDB Streams ESM | Lambda follows the lineage | Processes the parent shard before the child shards | no statement was found | DynamoDB Streams page | The position type in the child shard, where the position is kept |
| Flink new Kinesis source connector | Follows the lineage | Reads all records of the parent shard before going on to the child shard | Flink checkpoints and snapshots | Flink connector blog post | The position type in the child shard |
Flink legacy FlinkKinesisConsumer | Could not follow the lineage | — | Flink checkpoints and snapshots | Flink connector blog post | — |
| Amazon Data Firehose | no statement was found | no statement was found. It starts reading the source stream from LATEST | no statement was found | Firehose source page, Firehose integration page | How it handles resharding, as a whole |
| DynamoDB Streams with the Kinesis Adapter | The Adapter takes care of it. Without it, you call DescribeStream with a ShardFilter to look up the child shards | Processes the parent shard first | KCL lease table (in this article's reading) | DynamoDB Streams page, Adapter page | The turnover of about 4 hours (blog posts only) |
In the two rows for your own readers, "You decide" for the position in the child shard is this article's reading. The documentation states the order to read in when order is important, but not the position type.

7. On-Demand Mode and Service-Managed Record Distribution
This section covers two cases in which you do not call splits and merges yourself: the automatic resharding of on-demand mode, and service-managed record distribution from September 23, 2026.7.1 Automatic Shard Splitting in On-Demand Mode
The capacity mode page describes how an on-demand stream splits shards as follows.Kinesis Data Streams monitors traffic for each shard. When the incoming traffic exceeds 500 KB/s per shard, it
splits the shard within 15 minutes. The parent shard's hash key values are redistributed evenly across child shards.
On an on-demand stream, you cannot call
SplitShard, MergeShards, or UpdateShardCount (Section 3.3). In this article's reading, even when the service does the split, the state and lineage of the parent and child shards are the same as in Chapter 3, and the consumer handles them as in Chapters 4 through 6. What differs is that you do not decide when the split happens.7.2 The Developer Guide Has No Statement on Merging
For a statement that an on-demand stream merges shards,no statement was found in the developer guide and the API Reference. The search covered the capacity mode page, the resharding page, and the MergeShards and UpdateShardCount pages.The What's New post of July 24, 2026 announces that on-demand streams with On-demand Advantage mode can lower warm throughput. It describes how capacity goes down, but it does not use the word merge.
The stream adjusts to the requested capacity or the amount needed to support peak data ingest usage in the last
hour, whichever is higher.
The warm throughput blog post states that after warm throughput is lowered, the number of open shards goes down through merges.
After you set a lower warm throughput, the OpenShardCount decreases as Kinesis Data Streams merges shards to match
the requested capacity (subject to the one-hour peak safeguard).
This article writes about merging in on-demand mode as a statement of the warm throughput blog post. In this article's reading, it is safer to build the consumer of an on-demand stream on the assumption that it may read a child created by a merge, that is, a child shard that has an
AdjacentParentShardId.7.3 Service-Managed Record Distribution Does Not Move Existing Records
Service-managed record distribution was announced under the name Service-Managed Partition Keys in the What's New post of September 23, 2026, and the API sets it with the valueAUTO. The UpdateStreamRecordDistributionStrategy page describes AUTO as follows:AUTO – Amazon Kinesis Data Streams distributes records evenly across shards using service-managed algorithms, and
ignores any partition key and ExplicitHashKey that a producer provides.
About records already in the stream before the switch, the same page states:
New records that arrive after the change are distributed according to the new strategy. Records already in the
stream keep their original shard assignments and are not redistributed.
The service-managed page states that this mode can be used only with on-demand streams and that it does not guarantee order.
No ordering guarantees – Because the service distributes records across shards using internal algorithms, records
with the same business entity do not necessarily arrive at the same shard. Do not use this mode for workloads that
depend on ordering.
The service-managed consumer page states
The mechanism for traversing shards, handling resharding, and managing checkpoints remains the same. For a statement that names the lineage of parent and child shards or ChildShards, no statement was found; the search covered the service-managed page, the service-managed consumer page, the UpdateStreamRecordDistributionStrategy page, and the What's New post of September 23, 2026. In this article's reading, what this mode changes is which shard a producer's records go into, not where a consumer reads from after resharding. This site's What Breaks Ordering, Service by Service on AWS covers the effect on ordering.8. What This Setup Showed
This section writes up the results of one split and one merge on one stream, with the behavior of four kinds of consumers recorded. All of it is an observation from one run on November 2, 2026, and none of it is a documented promise. The observations do not override the statements in the sources quoted in Chapters 3 through 7.8.1 Setup
In us-east-1, a provisioned stream was created with two shards,shardId-000000000000 and shardId-000000000001, and a retention period of 24 hours, and four consumers read it at the same time.- The author's
GetRecordsreader: Reads each shard fromTRIM_HORIZON. When it receives a response in whichNextShardIteratoris null, it looks at theParentShardsof each element of that response'sChildShardsand starts reading fromTRIM_HORIZONonly the child shards for which it has received the end of every parent shard. - The author's enhanced fan-out reader: Calls
SubscribeToShardwith one registered consumer. When a subscription ends, it calls again withAFTER_SEQUENCE_NUMBERfrom the lastContinuationSequenceNumber. Its rule for moving on to child shards is the same as that of theGetRecordsreader. - One KCL 3.5.4 worker: Reads with
GetRecords, receives up to 10 records at a time, and waits 1.5 seconds after each batch. The record processor callscheckpoint()insideshardEnded. - One Lambda Kinesis ESM: Starting position
TRIM_HORIZON, batch size 10, batch window 0, andParallelizationFactor1. The function logs the shard and the sequence number range of each batch it receives and then waits about 2 seconds.
The producer kept writing records that carry a running number, with 64 partition keys. This section gives times in seconds from T+0, the moment the producer started writing. At T+301 seconds,
SplitShard split shardId-000000000000 at the middle of its hash key range, and at T+780 seconds, MergeShards merged the two child shards that the split had created. The producer was stopped at T+1,260 seconds and the consumers at T+1,567 seconds, and after that ListShards was called with each ShardFilter type. Last, with the consumers still stopped, UpdateShardCount changed the number of open shards from 2 to 3.This section shows sequence numbers by their last six digits only, as in
…208002. The author's two readers moved on to the child shards after reading the parent shards to the end because the author's code made them wait, not because of anything the service did. What these two readers observed is the content of the API responses.8.2 The Responses at the End of a Parent Shard
In the setup of Section 8.1, the lineage of the stream turned out as follows.shardId-000000000002 and shardId-000000000003 had a ParentShardId of shardId-000000000000 and an AdjacentParentShardId of null. shardId-000000000004, created by the merge, had a ParentShardId of shardId-000000000002 and an AdjacentParentShardId of shardId-000000000003. This is the same shape as the statement on the shared-throughput consumer page in Section 3.2.For each of the three parent shards, the
GetRecords reader received a response with zero records and a null NextShardIterator. For the split, the ChildShards of that response had two elements, shardId-000000000002 and shardId-000000000003, and the ParentShards of both held the single ID shardId-000000000000. For the merge, the end responses of both parent shards held one element, shardId-000000000004, and its ParentShards held two IDs, shardId-000000000002 and shardId-000000000003. This matches this article's reading in Section 3.2 that the element for a child created by a merge can hold the IDs of two parent shards.The enhanced fan-out reader also received, for each of the three parent shards, an event with zero records and a null
ContinuationSequenceNumber, and the ChildShards of that event had the same content as for GetRecords. The subscription ended 1 to 5 milliseconds after that event. Every subscription that had not reached the end of a parent shard ended after about 300 seconds. On whether a subscription closes at the end (Section 4.3: no statement was found), the subscription closed all three times in the setup of Section 8.1, but this is not a documented promise.In the merge, the readers received the ends of the two parent shards at different times. The
GetRecords reader received the end of shardId-000000000003 at T+809.4 seconds, waited for the end of shardId-000000000002, received it at T+811.1 seconds, and started reading shardId-000000000004 at T+812.0 seconds.The time when writes to the parent shard stopped did not coincide with the time when the stream returned to the
ACTIVE state. For the split, DescribeStreamSummary returned ACTIVE at T+306.6 seconds, and the ListShards response shortly after T+313 seconds already showed the ending sequence number of shardId-000000000000 and the two child shards. Even so, the producer's PutRecord responses kept returning shardId-000000000000 up to the record written at T+326.0 seconds, and the first record for which a child shard was returned was written at T+326.5 seconds. For the merge, too, the stream was ACTIVE at T+785.8 seconds, the last record for which a parent shard was returned was written at T+806.4 seconds, and the first record for which the child shard was returned was written at T+806.7 seconds. The ending sequence number of shardId-000000000000 that ListShards returned, …407490, was larger than the sequence number of the last record in that shard, …208002. In this article's reading, in the setup of Section 8.1, neither the return to ACTIVE nor the appearance of an ending sequence number in ListShards was a signal that the parent shard had been read to the end. The signals are the responses in Sections 4.1 and 4.3.8.3 Passing Sequence Numbers Across Shard Boundaries
Each time theGetRecords reader received the end of a parent shard, it ran three checks.- On the closed parent shard, it got an iterator with the sequence number of the last record as
AT_SEQUENCE_NUMBERand calledGetRecordsonce. All three times, it got back that last record and a non-nullNextShardIterator. - On the closed parent shard, it got a
LATESTiterator and calledGetRecordsonce. OnshardId-000000000000andshardId-000000000002, the response had zero records, a nullNextShardIterator, andChildShards. OnshardId-000000000003, it had zero records, a non-nullNextShardIterator, and an emptyChildShards. The reader did not make a second call with that iterator, so whether an end response would have come next is not known. - It passed the last sequence number of the parent shard as
AFTER_SEQUENCE_NUMBERtoGetShardIteratorfor the child shard. All three times,InvalidArgumentExceptioncame back.
For the split, the
InvalidArgumentException message was:Invalid StartingSequenceNumber. It encodes shardId-000000000000, while it was used in a call to a shard with
shardId-000000000002
As Section 5.2 records, no statement that the sequence number of a parent shard cannot be used on a child shard was found in the documentation. In the setup of Section 8.1,
GetShardIterator looked at which shard the sequence number belongs to and refused it for a different shard. In this article's reading, the position in a child shard is not carried over from the parent's sequence number; it is decided by a position inside the child shard.8.4 KCL 3.5.4 Leases
In the setup of Section 8.1, the KCL 3.5.4 worker calledshardEnded for shardId-000000000000 at around T+332 seconds. The worker's log states that at the same time, the ShutdownTask of shardId-000000000000 created the leases of the two child shards. The checkpoint value for both leases was TRIM_HORIZON, and their parentShardId was shardId-000000000000. The checkpoint of the parent shard's lease became SHARD_END, and its childShardIds held the two child shards. The record processors of the two child shards were initialized at T+351.9 seconds with a starting position of TRIM_HORIZON. This matches the HierarchicalShardSyncer code in Section 5.3.For the merge, at T+811.6 seconds, the
ShutdownTask of shardId-000000000002 created the lease of shardId-000000000004. At that moment, the lease of the other parent shard, shardId-000000000003, was also in the table, and its shardEnded came at T+814.0 seconds. The branch in Section 5.3 that postpones creating the lease when only one parent lease is in the table did not occur in the setup of Section 8.1. Before processing the child shard, the KCL logged this line.No need to block on parents [shardId-000000000002, shardId-000000000003] of shard shardId-000000000004
In this article's reading, the
checkpoint of both parent shards was already SHARD_END at that point, so BlockOnParentShardTask did not wait.About one minute after each parent lease became
SHARD_END, LeaseCleanupManager deleted it with the following reason. This matches the description of the default TRUE of cleanupLeasesUponShardCompletion in Section 6.3.as it has been completely processed and processing of child shard(s) has begun.
The attribute names in the lease table were
parentShardId, childShardIds, startingHashKey, and endingHashKey. As an implementation fact, DynamoDBLeaseSerializer in the KCL 3.5.4 source, retrieved on November 2, 2026, also defines CHILD_SHARD_IDS_KEY as childShardIds. Section 9.1 puts this next to the childShardId on the KCL tables page.8.5 The Lambda Kinesis ESM
In the setup of Section 8.1, the ESM invoked the function 1,382 times, and the records it delivered totaled 7,171. For all three child shards, the first batch of the child came after the last batch of its parent.- Split: The last batch for
shardId-000000000000was processed at T+327.0 seconds and held the last record of that shard,…208002. The first batches forshardId-000000000002andshardId-000000000003were both processed at T+346.0 seconds. - Merge: The last batch for
shardId-000000000002was processed at T+807.0 seconds and the last batch forshardId-000000000003at T+808.1 seconds, and the first batch forshardId-000000000004at T+827.0 seconds.
Only a few unprocessed records were left in the parent shards, however. When the first record went into a child shard, the parent records that the ESM had not yet processed numbered 5 (one batch) for the split and 4 (two batches) for the merge. The setup of Section 8.1 does not show how the ESM behaves when minutes' worth of records are left in the parent shard. As Section 6.4 records, no statement was found that the Lambda Kinesis ESM processes the parent shard first, and this observation does not stand in for such a statement.
8.6 The ShardFilter Results
In the setup of Section 8.1, after the consumers were stopped,ListShards was called with each ShardFilter type, with the results below. For the types that take a time, three times were used: before the split (T+240.9 seconds), between the split and the merge (T+543.3 seconds), and after the merge (T+905.8 seconds). In the table, shardId-000000000000 through shardId-000000000004 are shown by their last three digits.ShardFilter | Specified Value | Returned Shards |
|---|---|---|
FROM_TRIM_HORIZON | — | 000, 001, 002, 003, 004 |
AT_TRIM_HORIZON | — | 000, 001 |
AT_LATEST | — | 001, 004 |
AFTER_SHARD_ID | shardId-000000000000 | 001, 002, 003, 004 |
AT_TIMESTAMP | Before the split | 000, 001 |
FROM_TIMESTAMP | Before the split | 000, 001, 002, 003, 004 |
AT_TIMESTAMP | Between the split and the merge | 001, 002, 003 |
FROM_TIMESTAMP | Between the split and the merge | 001, 002, 003, 004 |
AT_TIMESTAMP | After the merge | 001, 004 |
FROM_TIMESTAMP | After the merge | 001, 004 |
In this article's reading, the three
FROM_TIMESTAMP results do not conflict with either of the two descriptions in Section 9.1. For the time between the split and the merge, shardId-000000000000, which closed before that time, was not returned, and the two shards that closed after that time and the two open shards were returned. ListShards accepted the name FROM_TIMESTAMP. The name FROM_TIMESTAMP_ID was not tested.8.7 The Short-Lived Shards of UpdateShardCount
In the setup of Section 8.1, with two open shards,shardId-000000000001 and shardId-000000000004, UpdateShardCount was called with a target of 3. After the stream was back in ACTIVE, ListShards showed five new shards, two of which were already closed.shardId-000000000005andshardId-000000000006: The parent isshardId-000000000004.shardId-000000000006was closed.shardId-000000000007andshardId-000000000008: The parent isshardId-000000000001.shardId-000000000007was closed.shardId-000000000009: TheParentShardIdisshardId-000000000006, and theAdjacentParentShardIdisshardId-000000000007.
The open shards became these three:
shardId-000000000005, shardId-000000000008, and shardId-000000000009. Judging from the lineage, one UpdateShardCount call led to two splits and one merge and created the two short-lived shards that the page in Section 3.3 describes. As this article's reading in Section 3.3 says, the shard counts before and after, 2 and 3, do not show these three operations.8.8 What This Setup Does Not Show
The setup of Section 8.1 does not show the following.- These are the results of one run, in one Region, at one time. Neither the time it took for writes to the parent to stop in Section 8.2 nor the
LATESTresponses in Section 8.3 were checked again. - Each of the four consumers read 7,171 records with no duplicates and no gaps, but the run had no failures or restarts. It does not show whether duplicates occur on a restart.
- The KCL setup is one worker that reads with
GetRecords. A KCL that reads with enhanced fan-out, several workers, leases moving between workers, and the case in which only one parent lease is in the table were not tried. - The Lambda ESM setup is only the case with a
ParallelizationFactorof 1 and a function that does not fail. An enhanced fan-out ESM, failures and retries, and the case with many records left in the parent shard were not tried. - On-demand mode, service-managed record distribution, Managed Service for Apache Flink, Firehose, and DynamoDB Streams were not tried.
9. Where the Sources Disagree and What They Do Not State
This section gathers the places where the sources disagree and the matters for which no statement was found.9.1 Where the Sources Disagree
On theFROM_TIMESTAMP type of ShardFilter, the sources disagree. The ShardFilter page states the following. The incldues in it is as in the original.FROM_TIMESTAMP - the response incldues all closed shards whose end timestamp is greater than or equal to the given
timestamp and also all open shards. Corrected to TRIM_HORIZON of the data stream if FROM_TIMESTAMP is less than the
TRIM_HORIZON value.
The description of
ShardFilter on the ListShards page states:If you specify the AT_TIMESTAMP or FROM_TIMESTAMP_ID type, you must also provide the value for the optional
Timestamp property. If you specify the AT_TIMESTAMP type, then all shards that were open at the provided timestamp
are returned. If you specify the FROM_TIMESTAMP type, then all shards starting from the provided timestamp to TIP
are returned.
The former says it returns the shards that closed after the given time and the open shards; the latter says it returns the shards from the given time to the tip. The latter writes the type name as
FROM_TIMESTAMP_ID in one place. In this article's reading, the results in the setup of Section 8.6 do not conflict with either description. This article does not use the observation to decide that either page is wrong.The names of the items in the KCL lease table differ between the version pages. The KCL 1.x and 2.x page writes
childshards and hashrange, and the KCL tables page writes childShardId, startingHashKey, and endingHashKey for KCL 3.x. As an implementation fact, the attribute name in the KCL 3.5.4 source, retrieved on November 2, 2026, is childShardIds, and the lease table in the setup of Section 8.4 also held it under the name childShardIds. This article sets this difference between the two documentation pages side by side without deciding whether it is a version difference or a difference in notation.On the partition key for on-demand streams,
the sources disagree. In its section on read and write exceptions, the capacity mode page writes you must specify a partition key with each record and says the same holds for on-demand mode. The service-managed page writes any partition key values provided in PutRecord or PutRecords API calls are ignored. for service-managed record distribution.On the phase of KCL 1.x, the table on the KCL lifecycle page gives Current phase as
Maintenance mode and End-of-support date as 2026-01-30. The top of the KCL 1.x and 2.x page writes, in the future tense, KCL 1.x will reach end-of-support on January 30, 2026. This article writes the two columns of the table as they are.9.2 Matters for Which No Statement Was Found
For the following matters,no statement was found. The scope of each search is given in the section noted.- The position type to start reading a child shard with (Section 5.2)
- That the sequence number of a parent shard cannot be used on a child shard (Section 5.2)
- That resharding is limited to one operation at a time (Section 3.3)
- That an enhanced fan-out subscription closes at the end of the shard (Section 4.3)
- That the Lambda Kinesis ESM processes the parent shard first (Section 6.4)
- Where the Lambda ESM keeps its position (Section 6.4)
- How Firehose handles resharding (Section 6.6)
- A statement in the developer guide or the API Reference that on-demand mode merges shards (Section 7.2)
9.3 What Only Blog Posts State and What Only the Implementation States
What only blog posts state, among the sources this article read: merging in on-demand mode (the warm throughput blog post), the turnover of DynamoDB Streams shards about every four hours (the DynamoDB and Lambda blog post and the DynamoDB Streams and KCL blog post), and the new Managed Service for Apache Flink connector following the lineage (the Flink connector blog post).Only the implementation states three things, in the KCL 3.5.4 source: the checkpoint of the lease created for a child shard at the end of the parent is
TRIM_HORIZON, the child shard does not wait if there is no lease for the parent shard, and the lease of a child created by a merge is not created while only one of the two parent leases is in the table.
10. How It Goes Wrong
This section lists, from Chapters 4 through 8, what happens when the position is handled wrongly. Each item gives what is done and then what happens.- A KCL record processor does not call
checkpoint()inshardEnded. Then thecheckpointof the parent shard's lease does not becomeSHARD_END, and in the implementation the child shard keeps waiting (Sections 4.4 and 5.3).
- Your own reader treats the response in which
NextShardIteratoris null like an empty response and discards it. Then it does not getChildShardsand has to look the child shards up again withListShards(Section 4.1).
- Your own reader starts reading a child created by a merge on the end of only one parent shard. Then, when order is important, it reads new records of the child before the unread records of the other parent shard (Section 5.1).
- Your own reader tries to get an iterator for the child shard with the last sequence number of the parent shard. For a statement on that use,
no statement was found(Section 5.2). In the setup of Section 8.3,InvalidArgumentExceptioncame back (Sections 5.2 and 8.3).
- Your own reader treats the stream's return to
ACTIVEas the signal that it may stop reading the parent shard. In the setup of Section 8.2, records kept going into the parent shard for about 20 seconds after the stream was back inACTIVE. The signals that the parent has ended are the responses in Sections 4.1 and 4.3 (Sections 4.1 and 8.2).
- The reader of a parent shard stops for more than 5 minutes and keeps using the same iterator. Then
ExpiredIteratorExceptioncomes back. If you confuse this with the 5 minutes of an enhanced fan-out subscription, you look for the cause in the wrong place (Sections 4.2 and 4.3).
- You assume from the statement on the DynamoDB Streams page that the Lambda Kinesis ESM also processes the parent shard first. That statement is about the DynamoDB Streams ESM (Section 6.4).
- You read the fact that
IteratorAgedoes not go down right after a split as the split not working. The IteratorAge re:Post article states that it does not go down right away because the existing records stay in the original shards (Section 6.4).
- You assume that switching to service-managed record distribution also spreads the records already in the stream evenly again. The
UpdateStreamRecordDistributionStrategypage states that they are not redistributed (Section 7.3).
- You assume that only splits happen in on-demand mode and do not handle child shards that have an
AdjacentParentShardId. The warm throughput blog post writes about merges (Section 7.2).
11. Frequently Asked Questions About Kinesis Resharding and Consumer Positions
This section answers questions that the readers in Section 1.1 tend to have, with the content of Chapters 3 through 8. Each answer comes from the sources named in the sections of those chapters.Q1. When a shard is split, does the consumer automatically continue from where it left off?
It depends on the type of consumer. The KCL chapter of the developer guide states that the KCL creates the leases of the child shards and processes the parent shards first. The KCL Java consumer page adds that checkpointing inshardEnded is mandatory. With your own GetRecords or enhanced fan-out reader, you write the code that receives the signal that the parent has ended and starts reading the child shards. No statement on how the Lambda Kinesis ESM handles parents and children was found in the Lambda Developer Guide (Section 6.4).Q2. What tells you that the parent shard has ended?
WithGetRecords, from the response in which NextShardIterator is null. ChildShards is returned only in that end response. With enhanced fan-out, from the event in which ContinuationSequenceNumber is null and the ChildShards of the end event. With the KCL, the record processor's shardEnded is called, and the record processor must write a checkpoint there.Q3. Is it enough to read the child shard from TRIM_HORIZON?
The documentation does not state the position type for a child shard. In the implementation, KCL 3.5.4 creates the lease of a child shard at the end of the parent withTRIM_HORIZON, and it did so in the setup of Section 8.4 as well (Section 5.3). In this article's reading, a child shard starts empty at the resharding, so inside the retention period TRIM_HORIZON points to the first record of the child shard. This is not a documented promise.Q4. Can the sequence number of the parent shard be used on the child shard?
The documentation states neither that it can nor that it cannot. ThePutRecord page states that sequence numbers are per shard in the stream and not across all shards. In the setup of Section 8.3, passing the last sequence number of the parent shard as AFTER_SEQUENCE_NUMBER for the child shard returned InvalidArgumentException.Q5. Does the Lambda Kinesis ESM process the parent shard first?
That statement was not found in the Lambda Developer Guide (Section 6.4). The statement that the parent shard is processed first is on the DynamoDB Streams page of the DynamoDB Developer Guide, and it is about the DynamoDB Streams ESM. In the setup of Section 8.5, the first batch of each child shard came after the last batch of its parent, but only a few unprocessed records were left in the parent shard, and the observation is not a documented promise.Q6. Are shards merged in on-demand mode?
The developer guide describes splits in on-demand mode, but no statement on merges was found there (Section 7.2). The warm throughput blog post states that when warm throughput is lowered, the number of open shards goes down through merges.Q7. Does switching to service-managed record distribution change how you read through a reshard?
In this article's reading, it does not. What this mode changes is which shard new records go into, and records already in the stream stay in their original shards, according to theUpdateStreamRecordDistributionStrategy page. The service-managed consumer page states that handling resharding remains the same, but no statement that names the lineage was found (Section 7.3).Q8. What happens if a split occurs while the KCL workers are stopped?
The KCL 1.x and 2.x page states that in KCL 1.14 and later and 2.3 and later, the leases of the child shards are created by the worker that processed the parent shard when it reaches the parent's end, or by the periodic scan ofPeriodicShardSyncManager. If the workers are stopped, the end of the parent shard is reached only after a worker comes back and reads the parent shard to the end. In the implementation (the KCL 3.5.4 source, retrieved on November 2, 2026; Section 5.3), if the parent shard's lease remains, the child shard is not processed until the parent shard's checkpoint becomes SHARD_END. In this article's reading, if unread records of the parent shard pass the retention period while the workers are stopped, those records cannot be read.12. Summary
When you split or merge shards, the parent shard closes and keeps its unread records for the retention period. A consumer learns that the parent has ended, decides where to read the child shard from, and keeps the position. Who takes care of these three things, and how far the documentation goes, depends on the type of consumer.With your own
GetRecords and enhanced fan-out, the signal that the parent has ended is in the API response, and you decide the position in the child shard and where to keep it. What the documentation states goes as far as reading the parent shards to the end before the child shard when order is important, and, also when order is important, reading both parent shards to the end for a child created by a merge. With the KCL, the KCL creates the child shard's lease, and the record processor has to checkpoint in shardEnded; the Javadoc in the KCL 3.5.4 source adds that otherwise the child shards make no progress (Section 4.4). Waiting, if the parent's lease exists, for the parent's checkpoint to become SHARD_END and starting the child shard's lease created at the end of the parent at TRIM_HORIZON are implementation facts of KCL 3.5.4 (Section 5.3).For how the Lambda Kinesis ESM handles parents and children, and for how Firehose handles them, no statement was found in the sources as of November 2, 2026 (Sections 6.4 and 6.6). For the DynamoDB Streams ESM, the statement that Lambda processes the parent shard first is in the DynamoDB Developer Guide. Merging in on-demand mode appears only in a blog post, and service-managed record distribution does not move existing records.
When you bring resharding into operation, first check which type your consumer is, and then check, separately, what the documentation states for that type, what only the implementation has, and what is not written anywhere.
13. References
- Amazon Kinesis Data Streams API Reference: GetRecords
- Amazon Kinesis Data Streams API Reference: GetShardIterator
- Amazon Kinesis Data Streams API Reference: SubscribeToShard
- Amazon Kinesis Data Streams API Reference: SubscribeToShardEvent
- Amazon Kinesis Data Streams API Reference: ChildShard
- Amazon Kinesis Data Streams API Reference: Shard
- Amazon Kinesis Data Streams API Reference: ListShards
- Amazon Kinesis Data Streams API Reference: ShardFilter
- Amazon Kinesis Data Streams API Reference: SplitShard
- Amazon Kinesis Data Streams API Reference: MergeShards
- Amazon Kinesis Data Streams API Reference: UpdateShardCount
- Amazon Kinesis Data Streams API Reference: PutRecord
- Amazon Kinesis Data Streams API Reference: UpdateStreamRecordDistributionStrategy
- Amazon Kinesis Data Streams Developer Guide: Reshard a stream
- Amazon Kinesis Data Streams Developer Guide: Complete the resharding action
- Amazon Kinesis Data Streams Developer Guide: Merge two shards
- Amazon Kinesis Data Streams Developer Guide: Develop shared-throughput consumers with the AWS SDK for Java
- Amazon Kinesis Data Streams Developer Guide: Develop enhanced fan-out consumers with the AWS SDK for Java
- Amazon Kinesis Data Streams Developer Guide: Choose the right mode to stream in
- Amazon Kinesis Data Streams Developer Guide: How service-managed record distribution works
- Amazon Kinesis Data Streams Developer Guide: Consumer considerations
- Amazon Kinesis Data Streams Developer Guide: Develop consumers with KCL in Java
- Amazon Kinesis Data Streams Developer Guide: Amazon Data Firehose consuming from service-managed streams
- Amazon Kinesis Data Streams Developer Guide: Use resharding, scaling, and parallel processing to change the number of shards
- Amazon Kinesis Data Streams Developer Guide: DynamoDB metadata tables and load balancing in KCL
- Amazon Kinesis Data Streams Developer Guide: KCL configurations
- Amazon Kinesis Data Streams Developer Guide: KCL 1.x and 2.x information
- Amazon Kinesis Data Streams Developer Guide: Monitor the Kinesis Client Library with Amazon CloudWatch
- Amazon Kinesis Data Streams Developer Guide: KCL version lifecycle policy
- AWS Lambda Developer Guide: Using Lambda to process records from Amazon Kinesis Data Streams
- AWS Lambda Developer Guide: Implementing stateful Kinesis Data Streams processing in Lambda
- Amazon DynamoDB Developer Guide: Change data capture for DynamoDB Streams
- Amazon DynamoDB Developer Guide: Using the DynamoDB Streams Kinesis adapter to process stream records
- Amazon Data Firehose Developer Guide: Configure source settings for Amazon Kinesis Data Streams
- AWS re:Post Knowledge Center: Decrease your Lambda function's IteratorAge metric
- AWS re:Post Knowledge Center: Change the number of open shards in Kinesis Data Streams
- AWS Big Data Blog: Introducing the new Amazon Kinesis source connector for Apache Flink
- AWS Big Data Blog: Scale down Kinesis Data Streams on-demand capacity with ODA warm throughput
- AWS Database Blog: Build scalable, event-driven architectures with Amazon DynamoDB and AWS Lambda
- AWS Big Data Blog: Process Large DynamoDB Streams Using Multiple Amazon Kinesis Client Library (KCL) Workers
- AWS What's New: Amazon Kinesis Data Streams now supports scaling down ingest capacity with warm throughput
- AWS What's New: Amazon Kinesis Data Streams announces Service-Managed Partition Keys for simplified data ingestion
- GitHub awslabs/amazon-kinesis-client: README (v3.5.4)
- GitHub awslabs/amazon-kinesis-client: CHANGELOG (v3.5.4)
- GitHub awslabs/amazon-kinesis-client: CHANGELOG (v1.x)
- GitHub awslabs/amazon-kinesis-client: HierarchicalShardSyncer.java (v3.5.4)
- GitHub awslabs/amazon-kinesis-client: BlockOnParentShardTask.java (v3.5.4)
- GitHub awslabs/amazon-kinesis-client: ShutdownTask.java (v3.5.4)
- GitHub awslabs/amazon-kinesis-client: ShardRecordProcessor.java (v3.5.4)
- GitHub awslabs/amazon-kinesis-client: DynamoDBLeaseSerializer.java (v3.5.4)
References:
Tech Blog with curated related content
Written by Hidekazu Konishi