Distributed Training Resilience on AWS - Where the Checkpoint Lives, How Far a Failure Sends You Back, and What Each Mechanism Requires

First Published: 2026-09-02
Last Updated: 2026-09-02

When running jobs with hundreds or even thousands of accelerators over weeks or months, failures inevitably occur. It's not uncommon for operations teams to manually restart jobs after each failure. And few within those teams can immediately state how much computation has been lost as a result.

The inability to answer this question isn't a matter of diligence. The answer isn't hidden somewhere within your configurations; it's determined by what state remained intact at the moment of failure. If the process can be stood back up while it is still alive, there is almost nothing to reload. If a process fails but the node it was running on remains operational, it may be possible to recover state from a healthy neighbor. However, if the entire node fails, everything after the last persisted checkpoint is lost.

In essence, the distance you need to roll back is determined not by the recovery procedure itself, but by where the state was stored.

This article examines two mechanisms offered by Amazon SageMaker AI's SageMaker HyperPod — managed tiered checkpointing and checkpointless training — from this perspective. While they are often presented as if one is simply faster than the other, the official material states a difference in the conditions under which each applies, not a ranking. One of them is responsible for durability and the other for recovery time, and neither one alone satisfies the requirement.

Furthermore, both mechanisms carry an unusual number of prerequisites. They are only compatible with clusters built on Amazon Elastic Kubernetes Service (Amazon EKS). A specific version of the training operator is required. Checkpointless training requires a minimum of two nodes. Dedicated container images are necessary. The supported frameworks and data formats are fixed as well. Whether you can adopt either approach is often determined by reviewing this list of requirements.

This article addresses hardware failures that happen on capacity you have already reserved, rather than price-driven interruptions such as a Spot Instance being taken back. The existing article Designing for Spot Interruptions on AWS covers how to design for recovery in interruptible capacity, discussing idempotency, the trade-offs in checkpoint granularity, and checkpoint placement, and includes a FAQ entry on distributed training. This article, however, focuses on the scenario where nodes do not get terminated but instead fail.

It also does not address the issue of checkpoint writing competing with collective communication for network bandwidth. That topic is covered in Section 9 of the existing article Elastic Fabric Adapter and the AWS Network Fabric. That article works through how training data reads, collective communication, and checkpoint writes all share one adapter, and how a checkpoint fired during a collective lengthens the very tail it was meant to shorten. This article will not repeat that explanation. However, as mentioned later, peer-to-peer recovery involves traffic flowing over the same fabric, adding one more flow to the ones the existing article already counts.

This article also does not cover how nodes are acquired in the first place. The existing article Gang Scheduling and Dynamic Resource Allocation on Amazon EKS details the allocation pathways using gang scheduling and Dynamic Resource Allocation (DRA). The existing article addresses the unit of allocation, while this article addresses the unit of recovery.

Finally, this article does not address inference. Even within the same HyperPod, the design that separates prefill and decode is covered in Disaggregated Prefill and Decode for LLM Serving on AWS, and serving models yourself on Amazon EKS is addressed in Self-Managed LLM Inference on Amazon EKS.

If the managed layer is enough, there is no need to come down this far. Customizing a model inside Amazon Bedrock is covered in Model Customization on Amazon Bedrock. That article ends by saying that once you leave Bedrock, the distribution strategy, the recovery from failures, and how far to roll back all become yours to decide. This article picks up from there. Conversely, the layer below that, instruction-level optimization, is covered in Programming AWS Trainium with the Neuron Kernel Interface.

This article does not discuss pricing. No amounts, and no unit prices. The official material connects fast recovery to cost by saying that it reduces the number of idle accelerators, but this article focuses solely on structural observations and does not delve into cost analysis. No benchmarking of my own was run for this article. Numbers that AWS has published are quoted with the attribution made explicit.

The specifications outlined in this article were verified against AWS's official documentation as of September 1, 2026. The prerequisites and version requirements are the sections most likely to change. Verify them against the primary sources before making a design decision.

One point of terminology is worth stating up front. The current official name is Amazon SageMaker AI. It was rebranded from Amazon SageMaker on December 3, 2024. For backward compatibility, however, the API namespaces, the AWS CLI commands, the AWS CloudFormation resource types, and the documentation URLs all remain sagemaker. This article will refer to the service as Amazon SageMaker AI, while retaining the original identifiers.

1. Failure Is the Steady State, Not the Exception
2. The Question Is Not How Long the Outage Was
3. The Recovery Ladder — How Far Back a Failure Sends You
4. Managed Tiered Checkpointing — Moving the Fast Tier into Cluster Memory
5. Where the Checkpoint Lives Changes Your Trust Boundary
6. Checkpointless Training — Recovering Without Reading a Checkpoint
7. Three Tracks, and the Sources Do Not Agree on How Many
8. The Two Mechanisms Are Not Alternatives
9. The Prerequisites Are the Adoption Decision
10. Who Notices the Failure in the First Place
11. What Sits Next to This and Is Not in This Article
12. Failure Modes and Anti-Patterns
13. Frequently Asked Questions
14. Summary
15. References

1. Failure Is the Steady State, Not the Exception

As systems scale, failures become the norm, not an exception. The AWS Machine Learning blog describes this situation as follows:

Because of the inherent all-or-none distributed synchrony across all ranks, even a loss of a single rank because of software or hardware faults brings the training workloads to a complete halt.

When a single rank fails, the entire job comes to a standstill. This isn't due to implementation flaws, but rather a consequence of the very definition of training, which requires synchronized gradient alignment. The same blog also discusses the relationship between scale and failure rates:

The gap between the system's theoretical maximum throughput and its actual productive output (goodput) grows larger with: increased frequency of failures (which rises with cluster size), longer recovery times (which scale with model size and cluster size), and higher costs of idle resources during recovery.

The key point is that two factors worsen simultaneously. As you increase the number of nodes, the frequency of failures increases. At the same time, the time required for recovery also increases with the number of nodes. The amount of time spent idle is the product of frequency and recovery time, so focusing on only one factor can lead to misinterpretations, particularly at larger scales.

AWS uses the term goodput. The blog defines it as:

goodput—the actual useful work accomplished in an AI training system compared to its theoretical maximum capacity

This article does not calculate a specific failure rate. Arguments based on estimating the average failure interval per node and multiplying those figures are unreliable, as the underlying assumptions are not publicly available. The crucial takeaway is that both the frequency of failures and recovery time degrade as the number of nodes increases. Beyond that point, the specifics will depend on observations within your own cluster.

This article addresses hardware and software failures that occur after capacity has already been reserved. Examples include GPU Xid errors, NVLink failures, driver crashes, and network timeouts. This is not the case where capacity is taken away from you. That topic is covered in the existing article Designing for Spot Interruptions on AWS. The FAQ in that article addresses distributed training as follows:

Can I use Spot for a distributed training job? Only if the job checkpoints and can resume with a different set of nodes.

This statement forms the basis of this article. The key difference is that with Spot instances, interruptions are triggered externally by price and demand, whereas in this article, interruptions are caused by failures. There is no warning, and no two-minute grace period.

2. The Question Is Not How Long the Outage Was

Incident reports typically focus on the number of minutes it took to restore service. However, in distributed training, this metric alone is insufficient. This is because the time it takes to recover and the amount of computation that needs to be re-executed are separate factors.

The AWS Machine Learning blog details the steps involved in recovering from checkpoints, outlining the process in a series of stages. Each stage must be executed in sequence, and progress cannot proceed to the next stage until the previous one is complete.

Each restart requires navigating a complex, multi-stage recovery process where every stage is sequential and blocking

The blog lists the following stages:

Stage	What Happens
Stage 1: Training job restart	The orchestrator detects the failure and terminates all processes on all nodes, restarting the job across the entire cluster.
Stage 2: Process and network initialization	All processes re-execute the training script from the beginning. This includes rank initialization, Python module loading, and communication backend construction.
Stage 3: Checkpoint retrieval	The system identifies the most recently saved checkpoint and retrieves it from persistent storage, loading the model weights, optimizer internal state, learning rate scheduler, and loop metadata.
Stage 4: Data loader initialization	The data-handling processes retrieve the data checkpoint and pre-fetch training data.
Stage 5: First step overhead	Initial overhead associated with the first step is incurred, including memory allocation, CUDA context construction, and CUDA graph compilation.
Stage 6: Lost steps overhead	Because steps calculated after the last checkpoint are lost, they must be recomputed.

The final two rows in this table represent the portion of the recovery process that is not typically included in the reported recovery time. The blog refers to Stage 5 as "first step overhead" (FSO) and Stage 6 as "lost steps overhead" (LSO). Although the job's processes are running, and it may appear to be recovered from a monitoring perspective, the cluster is actually re-performing work during the LSO phase.

Stage 2 and Stage 3 require varying amounts of time, as noted in the blog:

The process group initialization alone can take tens of minutes on large clusters.

This step can take tens of minutes or longer depending on cluster and model size.

Furthermore, these two stages differ only in the source of the data they access; both rely on retrieving information from external sources. Stage 2 retrieves connection information from other processes, while Stage 3 retrieves tensors from persistent storage. As the following sections show, both mechanisms go after that same act of fetching from outside.

One discrepancy inside the source material is worth writing down. The same blog, immediately following its listing of six stages, refers to them as five.

The five stages outlined above—termination and restart, process discovery and network setup, checkpoint retrieval, GPU context reinitialization, and training loop resumption—represent the fundamental bottlenecks in checkpoint-based recovery.

When counted again, this list of five does not include Stage 4, data loader initialization. Moreover, the AWS News Blog, which describes the same functionality, initially lists five stages, and those five do not correspond to the five listed above.

Traditional checkpoint-based recovery has these sequential job stages: 1) job termination and restart, 2) process discovery and network setup, 3) checkpoint retrieval, 4) data loader initialization, and 5) training loop resumption.

The content of these stages is the same across all the materials; only the way they are divided up differs. This article adopts the six-stage listing from the Machine Learning blog, because it is the only one that treats LSO as an independent stage. LSO, in essence, provides the answer to the question of how far back to revert.

3. The Recovery Ladder — How Far Back a Failure Sends You

The six stages outlined in the previous section are not necessarily traversed in their entirety each time a failure occurs. The point from which recovery begins depends on what resources remain active at the moment of the failure.

There are four places the state can live.

Where a Checkpoint Can Live, and What Each Placement Survives

First, there's the memory of the failed process itself. If the process can be restarted without a full shutdown, the CUDA context, compiler cache, and other resources remain intact, eliminating the need to reload anything. As noted in the AWS Machine Learning blog, this allows for in-process recovery:

The failed process stays alive (avoiding full process teardown), preserving the CUDA context, compiler cache, and GPU state, hence eliminating minutes of reinitialization overhead.

Second, there's the accelerator memory of healthy peers. In checkpointless training, each GPU maintains a replica of its model shard on a peer GPU. The process being recovered can then copy data directly from a healthy peer, rather than accessing storage.

Instead of periodically saving model state to centralized storage, each GPU maintains redundant copies of its model shards on peer GPUs. When a failure occurs, the recovering process doesn't load from Amazon S3. It copies state directly from a healthy peer over the high-speed Elastic Fabric Adapter (EFA) network interconnect.

Third, there's the CPU memory of the cluster. With managed tiered checkpointing, checkpoints are initially written to CPU memory and then replicated to adjacent nodes. As stated in the developer guide:

CPU memory serves as the primary tier to store model checkpoints. Secondary tiers include persistent storage options like Amazon S3.

It automatically replicates data across adjacent compute nodes for enhanced reliability. This replication strategy protects against single or multiple node failures while providing fast access for recovery operations.

The fourth place is Amazon Simple Storage Service (Amazon S3). This is the only one that outlives the cluster.

These four are not ordered by speed. They are ordered by how far each one reaches. A process's memory is lost when the process terminates. A peer's memory is lost if all nodes holding a copy of its shard simultaneously fail. The cluster's CPU memory is lost in a cluster-wide stop. Only Amazon S3 survives one.

The SageMaker HyperPod training operator bridges the top two rungs automatically, in what the same blog calls intelligent recovery escalation.

The operator implements intelligent recovery escalation: it first attempts in-process restart for failed components, and if that's not feasible (for example, because of container crashes or node failures), it escalates to process-level recovery. During a process-level recovery, instead of restarting the entire job when failures occur, the operator restarts only training processes, keeping the containers alive.

The Recovery Ladder, and What Escalates a Failure to the Next Step

The diagram illustrates that all the downward arrows fire because something was lost, rather than marking a transition from a good state to a bad one. If a process cannot be recovered in place, it drops one rung. If no healthy peer still holds the shard, it drops another rung. If the memory tier no longer holds that step, it drops all the way to the last rung.

How far back you go is the question of how many rungs down this ladder you fall. There are two ways to reduce the depth of that fall. One is to increase the probability that the higher rungs will survive. The other is to minimize the impact when a failure reaches the lower rungs. The former is achieved through checkpointless training, and the latter through managed tiered checkpointing.

The operator does more than walk the rungs. The blog also mentions that the operator broadcasts coordinated stop signals in the event of a failure.

When failures occur, the operator broadcasts coordinated stop signals to prevent cascading timeouts and integrates with the SageMaker HyperPod health-monitoring agent to automatically detect hardware issues and trigger recovery without manual intervention.

Preventing cascading timeouts is not part of recovery; it is a precondition for it. When one rank fails to respond, other ranks waiting for collective communication begin to time out one after another. If left unchecked, a failure that could have been handled on the first rung can escalate to a full system restart.

4. Managed Tiered Checkpointing — Moving the Fast Tier into Cluster Memory

Managed tiered checkpointing became generally available in September 2025. The What's New entry limits the supported orchestrator in the following way:

Managed tiered checkpointing is integrated with PyTorch's Distributed Checkpoint (DCP) and is available for SageMaker HyperPod clusters using the EKS orchestrator.

The developer guide also includes the same limitation. The introduction page for managed tiered checkpointing states that it handles PyTorch frameworks on Amazon EKS HyperPod clusters, and the configuration prerequisites begin with HyperPod clusters built on Amazon EKS. The AWS Machine Learning blog similarly states this, specifying the timeframe:

As of the time of this launch, managed tiered checkpointing is supported only on SageMaker HyperPod on Amazon EKS.

In other words, while SageMaker HyperPod itself can be configured using either Slurm or Amazon EKS, this feature is exclusively available on Amazon EKS. These two aspects are not contradictory. The existing article Disaggregated Prefill and Decode for LLM Serving on AWS also includes a similar limitation regarding features related to the inference side.

4.1 What the Cluster Installs When You Turn It On

This setting applies at the cluster level, not at the job level. The developer guide explicitly states this.

You must opt in to use managed tiered checkpointing.

To enable it, pass a TieredStorageConfig to CreateCluster or UpdateCluster. In the AWS CLI, this takes the following format:

aws sagemaker update-cluster \
    --cluster-name cluster-name \
    --tiered-storage-config '{ "Mode": "Enable" }'

The contents when Mode is set to Enable are best described in the API reference.

When set to Enable, the system installs a memory management daemon that provides disaggregated memory as a service for checkpoint storage. When set to Disable, the feature is turned off and the memory management daemon is removed from the cluster.

This daemon is the component that carves out a portion of the cluster's CPU memory to use as storage for checkpoints. The developer guide's removal procedure page even reveals its implementation details.

This removes the memory management daemon from your cluster. The daemon is implemented as a standard Kubernetes DaemonSet and follows standard Kubernetes lifecycle management.

The amount of memory allocated is specified using InstanceMemoryAllocationPercentage. The developer guide's configuration page states that the valid range for this value is between 20 and 100. The API reference and AWS CloudFormation resource specifications only describe this parameter as The percentage (int) of cluster memory to allocate for checkpointing., without specifying a range. Only the developer guide page provides this range.

4.2 What the Training Loop Has to Change

What the user's code has to do is move checkpoint reads and writes onto the PyTorch Distributed Checkpoint (DCP) framework and swap the storage implementation. First, add the library to the training image.

# Add this line to your training image Dockerfile
RUN pip install amzn-sagemaker-checkpointing s3torchconnector tenacity torch boto3 s3torchconnector

Next, replace the storage writer passed to DCP's async_save.

from torch.distributed.checkpoint import async_save, load
from amzn_sagemaker_checkpointing.checkpointing.filesystem.filesystem import (
    SageMakerTieredStorageWriter,
    SageMakerTieredStorageReader
)

# save_to_s3 decides whether this particular step also reaches the durable tier
checkpoint_config.save_to_s3 = training_step % s3_ckpt_freq == 0
storage_writer = SageMakerTieredStorageWriter(
    checkpoint_config=checkpoint_config,
    step=training_step
)

# Wait for the previous save before starting the next one. Section 4.3 explains why
# this line is required rather than merely advisable.
if future is not None:
    exc = future.exception()
    if exc:
        print(f"Failure in saving previous checkpoint:{str(exc)}")

future = async_save(state_dict=state_dict, storage_writer=storage_writer)

Pay close attention to save_to_s3. The frequency of writes to the memory tier and the frequency of persistence to Amazon S3 are set independently. The examples in the developer guide set the former to every 10 steps and the latter to every 50 steps. These two numbers decide how much work has to be redone at the third rung and at the fourth rung, respectively.

Where these two numbers live matters. Enabling the feature is a cluster setting, and the team that runs the cluster performs it. Conversely, the frequency of writes to each tier lives in the training script, a decision made by the team writing the job. Simply enabling the feature on the cluster side will not save anything. Similarly, modifying the script without enabling the feature on the cluster side will also result in no action. Because neither side does anything without the other, teams whose responsibilities span both should settle this point first.

The read side is symmetrical.

storage_reader = SageMakerTieredStorageReader(checkpoint_config=checkpoint_config)
load(state_dict, storage_reader=storage_reader)

SageMakerTieredStorageReader walks the tiers in order. The documentation for Ray describes this exploration process as follows:

load_checkpoint reads from tiered storage (cluster memory first, then Amazon S3). When a node is replaced and the job restarts via FailureConfig, recovery reads from the fast memory tier when available, avoiding a full Amazon S3 download.

From the user's code, there is no way to tell which tier the read came from. The same call can be the third rung or the fourth. The only way to tell which one it was is the logs. The developer guide provides instructions on how to access the library's logs from /var/log/sagemaker_checkpointing and mount them onto the host.

4.3 One Save at a Time

This mechanism has one constraint written into it. The page for Ray states it most plainly:

Only one async_save can be in flight at a time. The background threads perform collective operations that require all ranks to participate. Calling async_save again before the previous one completes causes a deadlock.

While the calls appear asynchronous, they internally involve collective communication. Therefore, initiating a new save operation before the previous one has finished will prevent all ranks from synchronizing, leading to a deadlock. The training loop example in the developer guide also demonstrates waiting for the previous future before initiating the next save operation.

This is not a performance suggestion; it is a constraint on correctness. It is also the first wall you hit when you try to raise the checkpoint frequency. The memory tier being fast does not mean the interval between saves can be shorter than a save itself takes.

5. Where the Checkpoint Lives Changes Your Trust Boundary

Where the checkpoint lives is a question about recovery and, at the same time, a question about your trust boundary. While often overlooked in feature introductions, the developer guide dedicates an entire page to this topic.

Managed tiered checkpointing uses Python's pickle module to deserialize checkpoint data stored in Amazon S3.

The document then outlines two key consequences.

Extended trust boundary: When using managed tiered checkpointing with Amazon S3, the Amazon S3 bucket becomes part of your cluster's trust boundary.

Code execution risk: Python's pickle module can execute arbitrary code during deserialization. If an unauthorized user gains write access to your checkpoint Amazon S3 bucket, they could potentially craft malicious pickle data that executes when loaded by managed tiered checkpointing.

The checkpoint bucket is not a place where data sits; it is a place where input gets executed. Every time you recover, whatever is present in that bucket is deserialized on the training nodes. Any entity with write access to the bucket effectively holds the same level of power as an entity capable of running code on your training cluster.

The same page lists several mitigation strategies: restricting permissions, implementing bucket policies, logging access, and carefully selecting bucket names. The last point refers to preventing attacks that attempt to hijack your bucket name.

Validate bucket names: Use caution with bucket name selection to avoid potential bucket hijacking.

The documentation also includes information regarding the network. Managed tiered checkpointing opens multiple ports on each compute node.

Managed tiered checkpointing enables network endpoints on each of your compute nodes on the following ports: 9200/TCP, 9209/UDP, 9210/UDP, 9219/UDP, 9220/UDP, 9229/UDP, 9230/UDP, 9239/UDP, 9240/UDP.

This page recommends maintaining the default restrictions.

By default, SageMaker's network configuration restricts access to these endpoints for security purposes. We recommend that you maintain these default restrictions.

While tiered memory was introduced for speed, it also creates new pathways for nodes to exchange state. The design of your security groups and network ACLs should be reviewed when enabling this feature.

6. Checkpointless Training — Recovering Without Reading a Checkpoint

Checkpointless training was announced on December 3, 2025. The What's New entry describes it as follows:

Checkpointless training maintains model training state across the distributed cluster, automatically swapping out faulty training nodes and using peer-to-peer state transfer for failure recovery.

While the name suggests it doesn't use checkpoints, a more accurate description is that it avoids reading checkpoints during recovery. It does not remove the need for persistence. This point will be revisited in Section 8.

6.1 Why the Minimum Is Two Nodes

Among the prerequisites, there's an item that might seem overly specific at first glance: the minimum cluster size is two nodes. The AWS Machine Learning blog explains the reasoning behind this requirement.

Minimum cluster size: Two nodes for peer-to-peer checkpointless recovery

Because recovery relies on peer-to-peer copying, a source is necessary. The mechanism does not hold in a configuration where the only place to put a shard's replica is the node itself. A single-node cluster failing to work here is not a limitation so much as a definition.

For the same reason, there's also an upper limit to the scope in which this mechanism can function. Recovery is only possible if healthy peers still hold replicas of the lost shard. If the nodes holding those replicas fail at the same time, you drop to a lower rung of the ladder.

6.2 What Each Track Shortens

Checkpointless training reduces time by optimizing specific stages, as discussed in Section 2. The developer guide describes these optimizations as three distinct tracks.

Checkpointless training is enabled via three optimization tracks that run in concert:

Track (as described in the developer guide)	Function	Stage(s) Optimized
Communication initialization improvements (NCCL and Gloo)	Distributes peer and ring information to eliminate bottlenecks in communication initialization.	Stage 2
Data loading optimizations	Reduces the time it takes to deliver the first batch after a restart.	Stage 4
Program restart overhead reduction	Reduces restart overhead and enables checkpointless recovery through process restoration on healthy nodes.	Stage 1, Stage 2, Stage 3, Stage 5, Stage 6

The first track addresses the issue of centralized communication initialization. As explained in the AWS Machine Learning blog:

A TCPStore is often used as a rendezvous point where all ranks check in to discover each other's connection information. When thousands of ranks try to contact a designated root server (typically rank 0) simultaneously, it becomes a bottleneck.

Instead, a system has been implemented where each rank independently calculates connection information for other ranks.

Instead of funneling all connection requests through a single root server, the system uses a symmetric address pattern where each rank independently computes peer connection information using a global group counter. Ranks connect directly to each other using predetermined port assignments, avoiding the TCPStore bottleneck.

Because the design stops funneling everything through a root, the single point of failure disappears along with the bottleneck. The blog writes that Process group initialization drops from tens of minutes to seconds.

The second track puts the data into shared memory.

Training data is mapped into shared memory regions that persist even when individual processes fail. When a node recovers, it doesn't reload data from disk but reconnects to the existing memory-mapped cache. The data loader state is preserved, helping to ensure that training continues from the correct position without duplicate or skipped samples.

This design also has a secondary benefit. According to the blog, data is now replicated only once per node. On a node with 8 GPUs, the traditional data loader previously required 8 copies of the data, but now only requires one, which reduces CPU memory consumption on the host.

The third track is what produces the first and second rungs of the ladder. It isolates failures to the process level, allowing healthy processes to continue running.

When a GPU or process fails, only the failed process executes an in-process recovery to rejoin the training loop within seconds, overcoming recoverable or transient errors. Healthy processes continue running without interruption.

In cases where the error is non-recoverable (such as hardware failure), the system automatically swaps the faulty component with a pre-warmed hot spare, enabling training to continue without disruptions.

The term pre-warmed hot spare encapsulates a key operational prerequisite for this mechanism. The speed of the swap depends on having a replacement component already warmed up and ready. As Section 10 shows, SageMaker HyperPod runs a long health check on every newly added node, and whether spares are on hand is what changes how that setting should be chosen.

6.3 The Recovery Path Runs on the Same Fabric

Peer-to-peer state transfer flows over the Elastic Fabric Adapter (EFA). The quote above says over the high-speed Elastic Fabric Adapter (EFA) network interconnect. The blog post also notes that the amount of data transferred is minimized.

The recovering node pulls only the specific shards it needs, further reducing transfer time.

This adds another item to the sequence described in Section 9 of the existing article, Elastic Fabric Adapter and the AWS Network Fabric. That article details how data loading, collective communication, and checkpoint writing all share the same adapter. In inference, key-value cache transfers are added as a fourth stream. The recovery path for checkpointless training represents a new addition to this flow. That article holds the analysis of how the bandwidth contention plays out, so this article defers to it.

7. Three Tracks, and the Sources Do Not Agree on How Many

The previous section called them three optimization tracks. This number varies depending on the document you consult.

Document	Counting Method
Developer guide (`HyperPod checkpointless training features`)	three optimization tracks
AWS News Blog (announcement dated December 3, 2025)	four core components
AWS Machine Learning blog (explanation dated December 15, 2025)	five components, and four tiers for the implementation process

A closer look reveals that the categorization differs.

Developer guide tracks	News Blog components	Machine Learning blog components	Implementation tiers
Communication initialization improvements	Collective communications initialization optimizations	Component 1: TCPStore-less/root-less NCCL and Gloo initialization	Tier 1
Data loading optimizations	Memory-mapped data loading that enables caching	Component 2: Memory-mapped data loading	Tier 2
Program restart overhead reduction	In-process recovery	Component 3: In-process recovery	Tier 3
(Same)	Checkpointless peer-to-peer state replication	Component 4: Peer-to-peer state replication	Tier 4
(Not counted as a track)	(Mentioned in the text)	Component 5: SageMaker HyperPod training operator	(Not counted as a tier)

There are two discrepancies. The developer guide folds in-process recovery and peer-to-peer replenishment into its third track. Only the Machine Learning blog counts the training operator as a separate component. The News Blog does not count the operator as a component, instead writing These components are orchestrated through the HyperPod training operator and thereby separating the roles.

None of these are incorrect. But if you remember there being three and then open another document, the count will not match. And when the count does not match, the first thing you suspect is your own misreading, which is what makes it slow to resolve.

Regarding the order of implementation, the Machine Learning blog provides clear guidance:

We recommend starting with Tier 1 and validating it in your environment. Add Tier 2 when data loading overhead becomes a bottleneck. Adopt Tier 3 and Tier 4 for maximum resilience on the largest training clusters.

These four tiers also correspond to the order of increasing code changes required. Tier 1 only requires adding two environment variables to the job specification. Tier 2 involves wrapping existing data modules. Tier 3 involves wrapping reusable code blocks with a wrapper. Tier 4 requires replacing NeMo's strategies and callbacks.

There is also a path that requires no code changes. As noted in What's New:

Checkpointless training is available in all AWS Regions where Amazon SageMaker HyperPod is available, and can be enabled with zero code changes for popular models like Llama and GPT OSS. For custom model architectures, minimal modifications are required for PyTorch-based workflows.

Changes are not required when using pre-defined recipes. The developer guide provides recipes for pre-training and fine-tuning both GPT OSS 120b and Llama 3 70b. If you are bringing your own model, you will need to implement those four tiers yourself. One caution about the Region named in that passage is covered in Section 9.

8. The Two Mechanisms Are Not Alternatives

These two approaches are often presented as an "old" method and a "new" method. However, this comparison misrepresents what each approach protects.

	Managed Tiered Checkpointing	Checkpointless Training
What it protects	The state persisting longer than the cluster's lifespan.	Training continuing despite failures.
Where the state is stored	Cluster CPU memory, replicated to adjacent nodes, and Amazon S3.	Accelerator memory of healthy peers.
Stages it acts on	Stage 3 and Stage 6. AWS describes the effect as faster recovery and less lost progress.	Stage 1 through Stage 6. AWS maps each component to the stages it optimizes.
In a cluster-wide stop	The state remains in Amazon S3.	Nothing remains.
Requirements for user code	Swap out DCP's storage writer and reader.	Use a recipe, or adopt the tiers you need out of the four.
Minimum configuration	Not specified.	2 nodes.

The fourth row of the table illustrates why these two approaches are not mutually exclusive. Checkpointless training remains robust as long as healthy peers remain. But in a cluster-wide stop, for example a job that is deliberately killed, a cluster that is rebuilt, or the simultaneous loss of the nodes holding a replica, nothing remains in peer memory. Therefore, durability ultimately relies on persistent storage.

Conversely, the AWS material describes managed tiered checkpointing as making checkpoint saves and recovery faster and the loss of progress smaller. It does not explicitly state that this approach shortens job restarts in Stage 1 or process group reconstruction in Stage 2. Optimizing those two specific steps is actually a focus of checkpointless training's first and third tracks.

The Machine Learning blog, describing the operator's role, writes on the assumption that the two coexist.

serving as the coordination layer that ties together initialization, data loading, checkpointless recovery, and checkpoint fallback mechanisms

checkpoint fallback mechanisms sits in that same sentence. When checkpointless recovery does not apply, the checkpoint is still there to fall back to.

Elastic training makes the relationship even clearer. Elastic training is a feature that dynamically scales the size of training jobs based on available cluster resources. The developer guide describes its implementation as follows:

Elastic training jobs can start with minimum compute resources required for model training and dynamically scale up or down through automatic checkpointing and resumption across different node configurations (world size).

Essentially, it takes checkpoints every time the scale is changed. The same page explicitly lists PyTorch Distributed Checkpoint as one of the integrated components. Announced on the same day as checkpointless training, and built on the same training operator, this feature relies on checkpoints.

Therefore, the question is not which one to use. There are two questions to consider. First, how quickly do you want to recover from a failure, assuming healthy peers remain? Second, how far back are you willing to go if no peer survives? The answer to the first question determines the rollout stage for checkpointless training, while the answer to the second determines how frequently data is persisted to Amazon S3.

The figures AWS has published are best read in the light of this same distinction. Regarding checkpointless training, the AWS Machine Learning blog states that internal testing showed recovery times reduced by 80 to 93 percent, and that clusters with thousands of accelerators achieved a goodput exceeding 95 percent. Concerning managed tiered checkpointing, the same AWS Machine Learning blog mentions validation across configurations ranging from hundreds to over 15,000 GPUs. Both are claims made by AWS themselves, and were not measured under identical conditions for comparison. There is no meaning in one set of numbers being larger than the other. They measure different quantities.

9. The Prerequisites Are the Adoption Decision

Both systems have numerous prerequisites. Reading through that list will generally determine whether or not adoption is feasible. It's best to review it beforehand rather than finding out after you've already started the design.

The following information was verified using primary sources as of September 1, 2026. The version requirements and the supported frameworks move, so take them again before deciding.

9.1 The Requirement Both Mechanisms Share

The sole common requirement for both mechanisms is that they operate within a SageMaker HyperPod cluster configured on Amazon EKS. For managed tiered checkpointing the developer guide and the What's New entry name Amazon EKS, and for checkpointless training the prerequisites section of the developer guide does the same.

While SageMaker HyperPod itself can be configured using Slurm, Slurm offers a separate fault tolerance system. It utilizes a combination of health monitoring agents and auto-resume functionality, allowing jobs to be resumed from the last checkpoint even after nodes are replaced. The limitation is that these two mechanisms are only available within Amazon EKS. For information on what Slurm provides within AWS, including managed pathways via the AWS Parallel Computing Service, see the existing article Batch and HPC Job Scheduling on AWS. This article focuses on where a job resumes from when it breaks while running in that environment.

9.2 What Managed Tiered Checkpointing Requires

Item	Description
Cluster	An Amazon EKS HyperPod cluster with sufficient CPU memory available for checkpointing.
Enablement	Pass a `TieredStorageConfig` to the `CreateCluster` or `UpdateCluster` API. It is disabled by default and requires an explicit opt-in.
Memory Allocation	The `InstanceMemoryAllocationPercentage` must be between 20 and 100.
Framework	PyTorch training workloads and DCP jobs. Both are supported.
Libraries	Include `amzn-sagemaker-checkpointing` and its dependencies in the training image.
IAM	Training Pods require write permissions to Amazon CloudWatch and Amazon S3. Configure this through the EKS OIDC setup.

9.3 What Checkpointless Training Requires

Item	Description
Training Operator	Version 1.2.0 or later. The operator documentation states that the latest HyperPod AMI is required for use. Updates are performed using the `UpdateClusterSoftware` API.
Minimum Node Count	2 nodes. Required for peer-to-peer recovery.
Recommended Instance Types	AWS recommends instance families ml.p5 and ml.p6.
Frameworks	Nemo, PyTorch, PyTorch Lightning
Training Data Format	JSON, JSONGZ (compressed JSON), ARROW
Container Image	A dedicated image is required. This image includes rootless NCCL initialization and peer-to-peer recovery functionality.

The container image needs a caution. The path to the image will not be provided in this article for two reasons.

First, the path varies across official documentation. The path listed in the Machine Learning blog and the path listed in the release notes of the developer guide differ, including the repository name and tag. Furthermore, the PyTorch version included in the path listed on the blog falls outside the supported range (2.4.0 to 2.7.1) specified on the training operator documentation page. While it is reasonable to treat the release notes, the dedicated page, as the definitive source, a path of this kind goes out of date the moment it is transcribed.

Second, the image is published under different accounts, depending on the AWS Region. The release notes list Region-specific paths. Including them here would require readers to search for the correct line corresponding to their Region within this article.

For the same reason, a list of supported Regions will not be included. There is, however, a discrepancy worth knowing about. The What's New entry states that checkpointless training is available in all AWS Regions where Amazon SageMaker HyperPod is available. In contrast, the release notes of the developer guide list the Regions where the container image is published and then explicitly state that it is not available in three opt-in Regions. It's crucial to understand that the availability of a feature and the presence of the image required for that feature in your Region are separate considerations. Decisions regarding adoption should be based on the information provided in the release notes.

9.4 The Version Requirements Move

Two version series appear in this section, and they belong to different things. The training operator is at v1.2.0 or later in the table above. The checkpointless training package has its own release notes, which document two versions to date: v1.0.0, released on December 3, 2025, and v1.0.1, released on April 10, 2026. The latter's updates address an issue where the fault handling thread incorrectly selected CUDA devices, and now correctly sets the device context using LOCAL_RANK. This update involves corrections related to the in-process recovery process itself.

The training operator also has supported versions. Kubernetes versions 1.28 through 1.33, and PyTorch versions 2.4.0 through 2.7.1 are listed as supported. While Kueue is not required, recommended versions are provided if you choose to use it.

Every one of these values goes stale the moment it is written down. This article provides a list of items to verify, and does not represent a confirmation of results.

10. Who Notices the Failure in the First Place

So far this article has discussed what happens after a failure is detected. Detection itself also decides which rung you land on. If a failure is noticed late, everything computed in the meantime has to be done again. If the detection method is flawed, it could lead to discarding nodes even when the failure is repairable.

The following describes the configuration when using Amazon EKS. The developer guide keeps the Slurm side of resilience on separate pages, built mainly around auto-resume. What this section cites is the Amazon EKS side.

SageMaker HyperPod's health monitoring consists of two components, as described in the developer guide:

1. Monitoring agents installed in your nodes, which include the Health Monitoring Agent (HMA) that serves as an on-host health monitor and a set of out-of-node health monitors.
2. Node Recovery System managed by SageMaker HyperPod.

10.1 Passive Checks and Active Checks

HMA monitors specific indicators for both GPUs and Trainium devices. For NVIDIA GPUs, these include DCGM policy violation notifications, errors appearing in the output of nvidia-smi, errors in the logs generated by the Amazon Elastic Compute Cloud (Amazon EC2) platform, and a verification of the number of GPUs. This GPU count verification is a simple but robust check that restarts the node if the actual count differs from the expected count. For AWS Trainium, the Neuron monitor output, the Neuron node problem detector output, the platform logs, and a count of the Neuron devices do the same job.

These are continuous, passive checks. The developer guide notes that active checks are also performed separately.

The above checks are passive, background health checks HyperPod runs continuously on your nodes. In addition to these checks, HyperPod also runs deep (or active) health checks during the creation and update of HyperPod clusters.

These deep health checks are performed during cluster creation and updates and can be requested at any time using the StartClusterHealthCheck API. These checks include, at the instance level, verification of the number of GPUs and NVLink connections, Level 4 DCGM diagnostics, validation of Neuron sysfs and actual training workloads (for Trainium), performance of collective communication on a single node, and measurement of EFA latency and bandwidth. At the cluster level, checks are performed to validate collective communication across multiple nodes.

These checks take a significant amount of time. The developer guide states that when deep health checks are enabled, every instance added to the cluster, whether at creation or through automatic node replacement, goes through roughly two hours of instance-level stress testing.

10.2 The Trade-Off Is Whether You Have Spares

The operational choice you face involves a trade-off, and the developer guide provides recommendations based on three scenarios, the key distinction being whether you have spare nodes available.

Scenario	Recommended Configuration
You have spare nodes, or can tolerate approximately 2 hours for health checks.	Enable deep health check throughout the cluster's lifetime.
You do not have spare nodes and want to bring replacement nodes online as quickly as possible.	Enable it only during cluster creation, and disable it afterward.
You do not have spare nodes and cannot wait 2 hours (e.g., a small cluster).	Disable it throughout the cluster's lifetime.

In all cases, automatic node recovery is enabled by default. Furthermore, the developer guide states:

If you want to resume the training job from a failure immediately, make sure that you have additional spare nodes as backup resources in the cluster.

The pre-warmed hot spare of Section 6 connects here. The ability to quickly replace a failed component with checkpointless training is possible because the replacement node has already completed its health checks and is ready. In a configuration without spares, acquiring and checking the new node swallows the speed the upper rungs bought you.

The automatic recovery setting is configured at the cluster level, choosing between Automatic or None. The developer guide recommends Automatic, and regarding None, it states:

If set to None, the health monitoring agent will label the instances when a fault is detected, but it will not automatically initiate any repair or recovery actions on the affected nodes. This option is not recommended.

10.3 What the Cluster Writes Down

When a fault is detected, SageMaker HyperPod records information in four ways: node labels, node taints, node annotations, and Kubernetes node conditions. The developer guide also specifies the number of annotations that are recorded.

Records up to 20 faults with timestamps that occurred on the node

The labels also have defined values. sagemaker.amazonaws.com/node-health-status can take four values: Schedulable, meaning the node has passed the basic checks and can accept workloads; Unschedulable, indicating the node is undergoing a deep health check; UnschedulablePendingReplacement, indicating the node requires replacement; and UnschedulablePendingReboot, indicating the node requires a reboot. Of these four, Schedulable, UnschedulablePendingReplacement, and UnschedulablePendingReboot are the results of the deep health check. Unschedulable represents a temporary state while the check is in progress. The developer guide also states that instances that fail the deep health check will be replaced.

A separate label carries the progress of the deep health check. sagemaker.amazonaws.com/deep-health-check-status can take three values: InProgress, Passed, and Failed. In addition, the fault-types and fault-reasons labels provide classifications and details about the detected faults, respectively.

The use of labels to represent the node's status allows Pods to select only healthy nodes using the node selector. The recovery design is directly linked to the scheduling design. However, the scheduling aspects are already covered in the existing article Gang Scheduling and Dynamic Resource Allocation on Amazon EKS, so this article stops here.

10.4 If You Are Not on HyperPod

If you are not using SageMaker HyperPod, the standard mechanisms within Amazon EKS handle the functionality that HyperPod would otherwise provide. The Amazon EKS Best Practices Guide sets the two side by side.

While the EKS Node Monitoring Agent with Auto Repair provides features like node health monitoring and auto-repair using standard Kubernetes mechanisms, SageMaker HyperPod offers targeted resilience and additional features specifically designed for large-scale ML training, such as deep health checks and automatic job resumption.

The same guide states that within Amazon EKS, job rescheduling and restarts rely on standard Kubernetes mechanisms and job restart policies. The two mechanisms discussed in this article sit on top of those.

11. What Sits Next to This and Is Not in This Article

Here are the neighboring topics that the existing articles already hold as their own subject. Without clear distinctions, both articles risk becoming diluted.

Bandwidth Contention. The existing article Elastic Fabric Adapter and the AWS Network Fabric, Section 9, addresses bandwidth contention. That article holds two things: checkpoint writes share an adapter with collective communication, and a checkpoint fired during a collective lengthens the very tail it was meant to shorten. That article quotes AWS on EFA and ENA sharing the same underlying resources, so that bandwidth used by one reduces what is available to the other. The only thing this article adds is that the peer-to-peer recovery path becomes one more flow on that same fabric.

Interruptible Capacity. The existing article Designing for Spot Interruptions on AWS, Section 7, discusses designing for Spot Instance interruptions. It covers idempotency coming before checkpointing, the risk of failure when the granularity is either too fine or too coarse, the requirement that a checkpoint outlive the instance, and the point that resuming is a first-class path rather than an error path. This article covers what changes once rungs appear inside that requirement to outlive the instance.

Node Allocation. The existing article Gang Scheduling and Dynamic Resource Allocation on Amazon EKS holds launching pods as a group, requesting devices through DRA, and the fork in the allocation path on Amazon EKS.

Inference. The existing article Disaggregated Prefill and Decode for LLM Serving on AWS, discusses the design of separating prefill and decode. The existing article Self-Managed LLM Inference on Amazon EKS, covers the design of providing models directly on Amazon EKS. Even on the same SageMaker HyperPod, the inference side belongs to the existing articles.

Amazon EKS Control Plane Configuration. The existing article Amazon EKS Control Plane Configuration, details what can be configured and what settings cannot be reversed.

Order of Feature Additions. The history of SageMaker features is documented in the existing article AWS History and Timeline regarding Amazon SageMaker. This article will not reorganize the timeline.

Elastic Training. As mentioned in Section 8, elastic training was announced on the same day as checkpointless training and runs on the same training operator, but it solves a different problem. While checkpointless training handles failures, elastic training focuses on available capacity. The latter adjusts the learning rate while increasing or decreasing the number of data-parallel replicas, maintaining a global batch size. Priority design and queue design belong to scheduling, so they go to the gang scheduling article above.

Reproducibility on the Training Side. This article does not address the question of whether the same numerical output is produced from the same input. The reproducibility of inference and its connection to the training process are covered in the existing article Reproducible LLM Inference. That article explicitly excludes reproducibility on the training side. Regarding the accuracy of checkpointless training, AWS states that it has verified that the loss values match bit-by-bit at each step. This is an AWS claim, and has not been measured in this article.

Quantization During Training. The topic of weight quantization is covered in the existing article LLM Weight Quantization on AWS, which explicitly states that it does not cover quantization during training. This article will also not delve into that area. It touches this article only in the form of what goes into a checkpoint; accuracy and format belong to that article.

Definitions of Terms. Definitions for the vocabulary around distributed training live in the existing article AI and Machine Learning Glossary for AWS.

The Kernel Layer. Even with the same accelerator, the layer where you decide the instructions, the memory, and the execution schedule yourself is covered in Programming AWS Trainium with the Neuron Kernel Interface. This article does not address instruction-level optimization. However, as seen in Section 10, deep health checks include Trainium inspection, so the layer beneath resilience does distinguish between accelerator types.

12. Failure Modes and Anti-Patterns

These are the shapes that looked easy to step into while reading the primary sources.

Planning these two features into a cluster built with Slurm. Both managed tiered checkpointing and checkpointless training are currently only supported on HyperPod clusters built with Amazon EKS. That SageMaker HyperPod can be built on Slurm and that these two features work on Slurm are two different statements. Slurm itself has distinct mechanisms, such as health monitoring agents and auto-resume, which can also resume jobs from the last checkpoint. Since the orchestrator is defined during cluster creation, fault tolerance considerations should be determined before the cluster is built.

Lowering the persistence frequency because checkpointless training is now in place. Recovery from peers only works for as long as healthy peers still hold the relevant shards. In a cluster-wide stop, nothing remains in peer memory. The persistence interval for Amazon S3 is still what decides how far back you go in the worst case. The AWS material itself is written on the assumption that the operator has a checkpoint to fall back to.

Firing async_save without waiting for the previous save to finish. Due to internal collective communication, not all ranks can synchronize, leading to deadlocks. This is not a performance issue. The job stops. The save interval cannot be shorter than the time it takes to perform the save itself.

Leaving the cleanup of a reported checkpoint to Ray when combining this with Ray Train. The developer guide recommends setting delete_local_checkpoint_after_upload=False. The reported checkpoint points at an Amazon S3 path managed by the tiered storage writer, so a delete by Ray removes the checkpoint from the durable tier.

Treating the checkpoint bucket as nothing more than a place to put data. On every recovery, whatever sits in that bucket is deserialized as a Python pickle on the training nodes. Anyone holding write permission can run code there. The bucket sits inside the trust boundary.

Treating the memory tier the same way as the durable tier. The CPU memory tier withstands single or multiple node failures through replication to adjacent nodes, but nothing remains after a cluster-wide stop. The presence of replication and the ability to outlive the cluster are separate concerns.

Copying the container image path straight out of the announcement blog. The path specified in the release notes does not match, and the PyTorch version included in the blog is outside the supported range for the training operator. Take the image path from the release notes, the page dedicated to this feature.

Reading the statement that the feature is available in all Regions as meaning that it works in your own Region. While What's New states that the feature is available in all HyperPod Regions, the release notes indicate that the container image is only available in a limited number of Regions. Check the reach of the feature and the reach of the artifact it needs as two separate things.

Trying peer-to-peer recovery on a single node. The copy needs a source. The minimum of two nodes is not a restriction so much as a consequence of how the mechanism is defined.

Leaving deep health checks enabled with no spare nodes on hand. A newly added node goes through roughly two hours of checks. In a configuration without spares, that time is added to the recovery time on every failure. For exactly this case, the developer guide offers the option of enabling the check only at cluster creation and turning it off afterward.

Writing figures published by AWS as if they were your own. The reduction in recovery time, the goodput values, and the number of validated GPUs are all claims made by AWS. The figures for the two mechanisms were also measured under different conditions, which makes them incomparable.

Remembering that there are three tracks, and then opening a different document. The developer guide lists three, the News Blog lists four, and the Machine Learning blog lists five. The content is the same; the categorization is different.

Monitoring recovery time alone. The moment the process comes up, monitoring shows the job as recovered, but the time spent recomputing the steps since the last checkpoint continues after that. The amount of work required to recalculate cannot be determined without examining the step number.

13. Frequently Asked Questions

Can managed tiered checkpointing be used with HyperPod clusters built using Slurm?

No, it cannot. The developer guide specifies that it is intended for HyperPod clusters built on Amazon EKS, and What's New also limits its use to clusters that utilize the Amazon EKS orchestrator. Checkpointless training has the same limitation. However, SageMaker HyperPod itself can be deployed using Slurm, and it offers a different set of resilience features, including a health monitoring agent and auto-resume.

If I implement checkpointless training, will I no longer need checkpoints?

No, you will still need them. Recovery from peer nodes is only possible as long as healthy peer nodes possess the relevant shards. In a cluster-wide stop, nothing remains in peer node memory. The AWS material itself is written on the assumption that the training operator has a checkpoint to fall back to. Elastic training, announced on the same day, is designed to take checkpoints and reload them whenever the scale is changed.

Which of the two should I put in first?

This article does not provide an answer to that question. The two features protect different things, and the official material does not rank them. There are two factors to consider when making your decision. First, how quickly do you want to recover from failures where healthy peer nodes remain? Second, how far back are you willing to go if no peer node survives? The former guides the implementation of checkpointless training, while the latter determines how frequently you should persist data to Amazon S3.

How many optimization tracks does checkpointless training have?

The number varies with the page you open. The developer guide lists three, the AWS News Blog announces four, and the AWS Machine Learning blog breaks them down into five. The content is the same; the difference lies in how they are categorized. The developer guide combines in-process recovery and peer-to-peer replenishment into a single track, while the Machine Learning blog treats the training operator as a separate element. As a guide to implementation, the Machine Learning blog provides a four-tier sequence and recommends starting with the first tier.

Why is the minimum requirement two nodes?

Recovery relies on a copy taken from a healthy peer, which means there has to be a source. The AWS Machine Learning blog explicitly states this requirement for peer-to-peer, checkpointless recovery.

Do checkpoints stored in memory persist if a node fails?

Yes, within limits. The developer guide states that these checkpoints are replicated to adjacent nodes, allowing them to withstand single or multiple node failures. They do not survive a cluster-wide stop. Durability comes from the Amazon S3 tier. Because reads across tiers are invisible to the user's code, the logs are what tell you which tier answered.

How frequently can checkpoints be saved?

The interval cannot be shorter than the time it takes for a save operation to complete. The developer guide explicitly states that initiating a new save before the previous async_save completes can lead to deadlocks, due to internal collective communication. The memory tier being fast does not mean the interval can be squeezed without limit. The trade-offs in checkpoint granularity are discussed in the existing article Designing for Spot Interruptions on AWS.

Are there any special considerations for the Amazon S3 bucket used for checkpoints?

Yes. The developer guide states that managed tiered checkpointing uses Python's pickle to deserialize checkpoints stored in Amazon S3, and that this bucket becomes part of the cluster's trust boundary. Any entity with write access can potentially execute arbitrary code on the training nodes. The same page highlights the importance of limiting permissions, using bucket policies, recording access, and carefully selecting the bucket name.

Does enabling managed tiered checkpointing change the network configuration of the nodes?

Yes. The developer guide states that listeners open on each compute node across several TCP and UDP ports. It also notes that these ports are initially restricted, and recommends maintaining those restrictions. The design of security groups and network ACLs should be reviewed when enabling this feature.

How much shorter will the recovery time be?

The AWS Machine Learning blog states that internal testing showed recovery times were reduced by 80 to 93 percent, and that clusters with thousands of accelerators achieved a goodput exceeding 95 percent. This is a claim made by AWS itself. No measurement of my own was made, so the number for your environment is yours to measure. The metrics you should measure include not only recovery time, but also the number of steps that need to be re-executed after the last checkpoint.

Does this article also apply when running distributed training on Spot Instances?

Partly, but the underlying assumptions are different. This article addresses hardware failures occurring on reserved capacity. The design considerations for situations where capacity is lost due to price and demand are detailed in the existing article Designing for Spot Interruptions on AWS. That article also notes that AWS names workloads that are tightly coupled between instance nodes as unsuitable for Spot.

14. Summary

This article organized the resilience of distributed training around a single question: how far back a failure sends you.

What is lost is not time. It is the delta from the last state that survived. When the AWS material breaks checkpoint-based recovery into stages, the last stage is recomputing the steps that were lost. The moment the processes come up, monitoring reads the job as recovered, and the cluster then spends more time doing work it has already done once.

There are four places the state can live, and they line up not by speed but by how far each one reaches. These are the memory of the failed process itself, the accelerator memory of healthy peers, the cluster's CPU memory, and Amazon S3. The process's memory disappears with the process. Peer memory is lost when the nodes holding a replica of that shard fail at the same time. The cluster's CPU memory is lost in a cluster-wide stop. Only Amazon S3 outlives the cluster.

The training operator bridges the top two rungs automatically, and below them the tiered storage read looks at the memory tier and then the Amazon S3 tier. Each rung down widens the range of work that has to be redone. The reason for dropping a rung is always that something on the rung above was lost, never a choice about performance.

The two mechanisms act on different parts of this ladder. Checkpointless training raises the odds that the upper rungs survive. Managed tiered checkpointing shrinks what has to be redone once a failure reaches the lower rungs. Neither mechanism alone can satisfy the requirements. Recovery from peers is not available in a cluster-wide stop, and AWS does not describe the memory tier as shortening job restarts or process group reconstruction. Elastic training, announced on the very same day, takes a checkpoint and reads it back every time the scale changes, which is the plainest evidence of this relationship.

Where the checkpoint lives also decides where the trust boundary runs. The developer guide explicitly states that the Amazon S3 bucket becomes part of the cluster's trust boundary, and that pickle deserialization happens during recovery. Even the memory tier, introduced for speed, opens new points of communication between nodes.

The list of prerequisites largely determines whether adoption is feasible at all. Both options only work with HyperPod clusters built on Amazon EKS. Checkpointless training has version requirements for the training operator, requires a minimum of two nodes, necessitates a specific container image, and has limitations regarding supported frameworks and data formats. Furthermore, the paths and Regions for the container images are inconsistent across official documentation. The reach of the feature and the reach of the artifact it needs have to be checked separately.

The detection layer sits in front of the ladder, not on it. Passive health monitoring runs continuously, and the deep health check, when enabled, takes roughly two hours on every instance that joins the cluster. Whether to leave it enabled turns on whether you have spare nodes. A failed component can only be replaced immediately if a replacement is already available and has completed its testing.

The count itself varies with which page of the official material you open. The developer guide lists three tracks, the News Blog four components, and the Machine Learning blog five components plus a four-tier adoption path. They all describe the same thing, and only the division differs. It is better not to settle for a single page of the primary sources.

Checkpoint writes competing with collective communication for bandwidth is the subject of the existing article Elastic Fabric Adapter and the AWS Network Fabric. And the recovery paths discussed in this article also operate on the same fabric. The options that stay inside the managed layer are described in Model Customization on Amazon Bedrock, while the layer below, where you decide the instructions, the memory, and the execution schedule yourself, is discussed in Programming AWS Trainium with the Neuron Kernel Interface.

15. References

References:
Tech Blog with curated related content

Written by Hidekazu Konishi

hidekazu-konishi.com