Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

## Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

Silent data corruption (SDC) happens when hardware, such as a GPU, returns a wrong but plausible number and raises no error. In distributed training, the wrong number can flow into the gradients and weights of every replica. Loss can still look normal for hours, and by the time a team notices an unexplained divergence or a weaker model, the compute is spent. It happens at scale: during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, and those were only the cases that interrupted the job and were traced to SDC. Amazon SageMaker HyperPod already replaces nodes that report a fault: the health-monitoring agent watches for GPU errors, deep health checks stress-test nodes, and automatic node recovery swaps out bad hardware. SDC slips past all of these features, because the hardware never reports anything. Detection has to come from the training job itself. In this post, we add three checks to a PyTorch DistributedDataParallel (DDP) training loop to detect SDC. When a check fires, we map the suspect rank to its node, contain the corruption, and have SageMaker HyperPod replace the node. Each check covers a different place where a fault can land: A non-finite gradient check catches NaN, Inf, and overflow in a GPU’s forward or backward pass, on the step it happens, and NaN or Inf that the gradient all-reduce introduces. A ring replay audit catches finite wrong values. On audit steps, every GPU recomputes a micro-batch from another node and compares the loss bit for bit and the gradients by fingerprint. A weight fingerprint comparison catches corruption in the optimizer step and in stored weights, by comparing every replica’s weights after the update. We tested the checks with software-injected faults on a SageMaker HyperPod cluster with 4 ml.p5.48xlarge instances (32 NVIDIA H100 GPUs), and repeated every test on three instances, the minimum the checks need. The full code of this solution is available in the accompanying GitHub repository . How silent data corruption happens and why the usual checks miss it SDC comes from manufacturing defects (marginal transistors that slip past vendor screening) and silicon aging in a specific compute unit, most often a multiply-accumulate lane in a tensor core, which is exactly the hardware model training hammers. That marginal circuit computes correctly most of the time and wrong only under stress: high temperature, voltage droop, or specific operand patterns. Because the arithmetic logic path has no ECC or parity (unlike memory), the wrong result, say 4.0001 instead of 4.0000, is never flagged, and it flows through activations and gradients into every replica’s weights. The reason it is so hard to catch is that SDC is in the compute path, and every integrity check you already have guards a different boundary : Check What it guards Why it misses SDC ECC memory Catches bit flips in memory, not wrong math. There is no ECC on a tensor-core result. Xid errors (NVRM) GPU driver/hardware faults The GPU raises an Xid when it detects a fault. SDC by definition raises none, so nothing lands in /aws/sagemaker/Clusters/ HMA logs. DCGM deep health check hardware faults Misses defects that need your workload’s operands, thermal state, or timing. Runs on an idle node, outside your job. NVLink / PCIe / EFA checksums transport Verify bytes arrived as sent, not that they were computed correctly. Checkpoint checksums storage round-trip A poisoned-but-consistent checkpoint checksums fine. The way to catch SDC is to check the result of the computation itself. Solution overview The loop has four steps. Your training job does the first two, and the last two use Kubernetes and SageMaker HyperPod features: Step 1. Detect: Three checks in your training loop name a suspect GPU by comparing data-parallel replicas. Step 2. Localize: On SageMaker HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS), the Kubernetes node name contains the instance ID, so mapping a suspect rank to a node is a lookup. Step 3. Contain and correlate: Stop the job, cordon the node, and look at its GPU metrics in the SageMaker HyperPod observability dashboards. Step 4. Confirm and recover: Confirm when the alarm is ambiguous, replace the node after review, and resume from a checkpoint saved before the suspect window. Figure 1: A SageMaker HyperPod EKS cluster with three p5 nodes running training pods and SDC checks. Node 1 raises an alarm for rank 13, and the proposed steps cordon the node, check GPU metrics, replace it, and restart training It helps to see the loop as layers, each adding something different: Layer Source What it adds Catches truly silent SDC? 1. Non-finite gradient check Your training script Names a GPU whose local gradient is NaN or Inf, before the all-reduce, and stops the step if the reduced gradient is NaN or Inf Only faults that become NaN, Inf, or overflow 2. Ring replay audit Your training script Every GPU recomputes another node’s micro-batch and compares the loss bits and gradient fingerprints Yes, for finite compute-path faults that change the loss or a gradient, on audit steps 3. Weight fingerprint comparison Your training script Names a replica whose weights differ from a strict majority after the update Yes, for optimizer-step and stored-weight faults on one copy. Blind to compute-path faults and to sharded weights 4. Localization Your training script, using SageMaker HyperPod node names Maps a suspect rank to an instance ID from the Kubernetes node name n/a: it localizes 5. DCGM correlation HyperPod observability add-on The suspect GPU’s DCGM metrics around the alarm It confirms faults only where they also appear in hardware counters. Clean telemetry doesn’t clear a node. 6. Deep health check StartClusterHealthCheck (DCGM L4) An on-demand hardware diagnostic on the suspect node It catches hardware defects its own test patterns trigger. Triage, not proof 7. Repeat-offender history A query you build over retained alarms and telemetry Nodes that alarm repeatedly Partly: it surfaces nodes worth checking early Layers 1–3 are the only layers that detect silent SDC, and they live in your training code. Layer 4 turns their alarm into an instance ID, and the HyperPod layers corroborate the alarm and close the loop. Prerequisites To follow along, you need: A SageMaker HyperPod cluster orchestrated by Amazon EKS, with at least three GPU nodes. The ring audit needs three different nodes for every comparison it settles, and the weight comparison needs three copies of each weight to name the odd one out. See Orchestrating SageMaker HyperPod clusters with Amazon EKS . A PyTorch DDP training job in which every replica runs its own optimizer step, with a data pipeline that can rebuild any rank’s micro-batch for a given step, such as a seeded sampler. Checkpoints on shared storage, Amazon FSx for Lustre or Amazon Simple Storage Service (Amazon S3). Replacing a node destroys its root and secondary volumes, including anything on local NVMe. The SageMaker HyperPod observability add-on for the correlation step. Turn on advanced DCGM metrics if you want ECC, NVLink, and throttling counters. Access to the cluster through kubectl , and AWS Identity and Access Management (IAM) permissions for sagemaker:StartClusterHealthCheck and sagemaker:BatchReplaceClusterNodes . See Manually quarantine, replace, or reboot a node . Step 1: Detect with three checks in your training loop Data parallelism trains several copies of the model, called replicas, at the same time. Each replica trains on a different slice of the batch. After each backward pass, an all-reduce averages the replicas’ gradients, and every replica applies that same average in optimizer.step() . Here’s one weight across three replicas, with a learning rate of 0.1: replica 0 replica 1 replica 2 weight at start 0.500 0.500 0.500 same local gradient 0.20 0.40 0.30 different (different data) averaged gradient 0.30 0.30 0.30 same, after the all-reduce weight after step 0.470 0.470 0.470 same Local gradients differ, because each replica sees different data. And after the optimizer step, the weights must be identical, bit for bit. Where a fault lands decides which check can see it. If replica 1 computes a local gradient of 4.00 instead of 0.40, the all-reduce spreads that error to every replica: all three apply the same wrong update, and their weights still match. Only replica 1’s local gradient looked different, and only before the all-reduce. If instead replica 1 corrupts a weight during or after the optimizer step, its weights no longer match the others. There’s one exception: a NaN or Inf in one replica’s weights reaches every replica through the next all-reduce, and after that the replicas agree again. A fault inside the all-reduce itself, such as a GPU that adds a partial sum wrong, also reaches every replica identically. So compute-path faults must be caught before the optimizer step, while the bad gradient can still be discarded, and faults in the update or in stored weights after it. Finite wrong values need recompute: comparing gradient norms across replicas catches only NaN and Inf (see Why not compare local gradient norms? ). That gives three checks on every DP rank, in a fixed order within each training step: Non-finite gradient check – Before the all-reduce, flag any rank whose local gradient is NaN or Inf. This catches values that blow up in a GPU’s forward or backward pass, on the step they happen, and names the GPU. After the all-reduce, check the reduced gradient for NaN or Inf too, before the update. Ring replay audit – On audit steps, before the optimizer step, every rank recomputes the micro-batch of a rank on another node. It compares the loss bit for bit and, with deterministic kernels, every parameter’s gradient fingerprint. This catches finite wrong values in the compute path. Weight fingerprint comparison – After the optimizer step, every N steps and before every checkpoint, compare every replica’s weight fingerprints. This catches faults in the optimizer step and in stored weights. The code in this step is for PyTorch DDP. For tensor, pipeline, or context parallelism, see Beyond DDP . Order of execution Figure 2: Flowchart of one training step. Check 1 (NaN/Inf gradients, before and after the all-reduce) and check 2 (ring replay, on audit steps) run before the optimizer step and stop the step if they fire. Check 3 (weight fingerprints) runs after it and blocks the checkpoint Why not compare local gradient norms? A natural first design, and the one we started with, flags a rank whose local gradient norm is more than k median absolute deviations (MADs) from the median of that step. We recorded every rank’s local gradient norm on every step of the three fault-free runs on 4 instances (22,800 steps, see Fault-free runs ) on 32 H100s training a 953-million-parameter model, and the test doesn’t work: The healthy spread is too wide – Each rank sees different data, so healthy local norms already reached 224 MADs in the first 800 steps, and 1,240 MADs over all healthy steps. A quiet threshold sees nothing finite – At k = 336, 1.5 times the highest score in the first 800 healthy steps, it raised 11 false alarms in the remaining 22,000 healthy steps. And no finite fault came close: the faulty rank’s score never exceeded 6.9, even with one activation multiplied by 10,000. Compare by fingerprint Checks 2 and 3 compare tensors across GPUs. Sending whole tensors would cost too much, so each tensor is reduced to a fingerprint: two 64-bit sums of its raw bit patterns, one plain and one weighted by position. Any change to one or two elements of a tensor changes the fingerprint, for tensors with fewer than 2³² elements of up to 4 bytes each. That covers every weight fault we injected. Larger changes could, in principle, cancel out, so equal fingerprints are strong evidence of equal tensors, not proof. The position-weighted sum matters: in our tests, a swap of two weight values and a compensating change to two values left the plain sum unchanged, and only the weighted sum caught them. _INT_VIEW = {1: torch.uint8, 2: torch.int16, 4: torch.int32, 8: torch.int64} @torch.no_grad() def fingerprints(tensors, chunk=1 = 2) ambiguous = [p for p in pairs if not set(p) & set(culprits)] return culprits, ambiguous def confirm(module, compute_loss, rebuild_batch, step, pair, table, stats, offset, with_grads): # … the third GPU (target + offset) and the target replay the target’s micro-batch again third_r, self_r = v.cpu() orig, peer = table[ti, 0], table[ri, 1] if torch.equal(third_r, orig): return {“pair”: pair, “third”: third, “suspect”: replayer, “kind”: “replayer computed wrong”} if torch.equal(third_r, peer): kind = (“persistent” if torch.equal(self_r, orig) else “transient” if torch.equal(self_r, third_r) else “unclear”) return {“pair”: pair, “third”: third, “suspect”: target, “kind”: kind} return {“pair”: pair, “third”: third, “suspect”: None, “kind”: “unattributable”} The confirmation is only independent if the three GPUs are on three different nodes. check_topology verifies that for every pair before training starts, from each pod’s NODE_NAME , and raises an error otherwise. Choose the audit mode: Default kernels: Compare the loss only. This needs a forward pass whose loss is bitwise reproducible, which held in our tests: healthy replays matched the original loss bit for bit. Gradients can’t be compared this way, because attention’s default backward pass isn’t deterministic. This mode catches forward-pass faults that change the loss. Deterministic kernels ( deterministic_audits=True ): The whole audit step runs with deterministic algorithms and the math attention kernel. Gradient fingerprints then match too, so the audit also catches backward-only faults. An op with no deterministic implementation raises an error instead of producing a false alarm. On our model, a deterministic audit step cost 2.4 times a default one. Never replay a suspect’s work on the suspect alone. A GPU with a persistent defect can reproduce its own wrong answer. Check 3: Weight fingerprints, after the optimizer step After optimizer.step() , every replica’s weights must be identical, so there’s no tolerance to tune. Each rank fingerprints every weight tensor and gathers everyone’s fingerprints. A rank is named only when its copy of a tensor differs from a strict majority, more than half of the copies. With two copies, or a split with no strict majority, the comparison reports a disagreement without naming any rank. def find_divergent_ranks(params, dp_group): local = fingerprints(params).flatten() # … all-gathers every rank’s fingerprints into table: [dp_size, num_tensors, 2] differs = (table != table[0]).any(dim=2).any(dim=0) if not differs.any(): return {“diverged”: False, “suspects”: []} suspects = set() for k in differs.nonzero().flatten().tolist(): copies = [tuple(row) for row in table[:, k].tolist()] value, count = Counter(copies).most_common(1)[0] if 2 * count ” valueFrom: fieldRef: fieldPath: spec.nodeName At job start, log a table that maps every rank to its node, instance, and GPU UUID: import os import torch import torch.distributed as dist node = os.environ[“NODE_NAME”] # “hyperpod-i-0123456789abcdef0” instance_id = node.removeprefix(“hyperpod-“) # “i-0123456789abcdef0”, what the Replace API takes gpu_uuid = “GPU-” + str(torch.cuda.get_device_properties(torch.cuda.current_device()).uuid) info = {“rank”: dist.get_rank(), “node”: node, “instance_id”: instance_id, “gpu_uuid”: gpu_uuid} table = [None] * dist.get_world_size() dist.all_gather_object(table, info) if dist.get_rank() == 0: print(“rank_map”, table) # log at every job start If a job is already running without the downward API, you can map a pod to its node from outside with kubectl get pod -o jsonpath='{.spec.nodeName}’ . Step 3: Contain the damage and check the GPU metrics When an alarm fires, stop the job first. Then stop new work from landing on the suspect node by cordoning the node while you investigate: kubectl cordon # marks the node unschedulable; existing pods keep running Then open the GPU health dashboard in the SageMaker HyperPod observability add-on, filter to the suspect node and GPU, and look at the time the alarm fired. A rise in correctable ECC errors, PCIe replays, throttling, or temperature on that GPU raises confidence that the hardware is at fault. A clean panel doesn’t clear the node: pure SDC often leaves no trace in hardware telemetry. Step 4: Confirm, then replace the node With the Step 1 checks running, an alarm already names a suspect GPU. When you see a symptom without an alarm, such as a loss spike, or want to rule out software before you replace anything, start with the cheap explanations. Most loss spikes aren’t SDC, and chasing SDC on a data bug wastes days: Loss anomaly / suspected corruption – Reproduces with the same seed on DIFFERENT nodes? -> deterministic bug, not SDC – Goes away, or follows one node, when you move the job? -> strong SDC candidate – DP replicas that must be bitwise identical disagree? -> strong SDC signal – One rank’s step replayed on another node gives different bits? -> strong SDC signal – One rank’s pre-all-reduce gradient is NaN or Inf? -> SDC candidate (use the first step it fired) – Fleet-wide and gradual? -> likely recipe or optimizer – Started at a data, config, or library change? -> usually software, but not proof The test that separates SDC from software is whether the problem follows the hardware: A reproducible failure doesn’t rule out SDC – Meta pinned a failing workload to one core and got the same wrong answer every time for specific inputs. Neither does a failure that starts at a software change – Google traced one incident to a correct library update that began using rarely used instructions on defective cores. Don’t expect the global loss to point anywhere – A forward fault first changes the faulty rank’s own loss, and the all-reduce then spreads its gradient to every replica. Per-rank checks and replays give you the locality. Decide what the alarm means Alarm Meaning Suggested action check == “weights” , suspects == [r] Rank r’s copy differs from a strict majority after the update Keep r’s node cordoned. Replace it after review check == “weights” , suspects is None Copies disagree, but no strict majority Ambiguous: keep the nodes involved cordoned. Check DCGM and run deep health checks culprits == [r] Both of r’s ring comparisons mismatched. This assumes at most one faulty GPU among the GPUs that replay each other (the GPUs with the same local index). With two, a healthy GPU can be named Keep cordoned. Replace after review, or after the fault repeats Verdict persistent The target reproduced its own wrong answer, and a third GPU agreed with the replayer Keep cordoned. Replace after review Verdict transient The target’s retry came out right and agreed with the third GPU. An intermittent fault can look the same Keep cordoned until an audit with the node as the target, or a deep health check, passes. Replace it if it happens again Verdict replayer computed wrong The replaying GPU, not the target, was wrong Treat the replayer as the suspect, and confirm again nonfinite set, no mismatched pairs NaN or Inf in a backward pass that the loss doesn’t show, or a gradient norm too large for float32 (above about 1.8 × 10¹⁹) Ambiguous: audit with the suspect as the target, or check DCGM and run a deep health check check == “reduced gradient” The gradient was NaN or Inf after the all-reduce, although every rank’s local gradient was finite. No rank is named Ambiguous: keep the job stopped, check DCGM and run deep health checks on every node in the job, and resume from a checkpoint saved before the alarm step Verdict unattributable or unclear The three GPUs don’t line up Ambiguous: check DCGM and run deep health checks on the nodes involved In production, keep a human approval on every replacement until you have evidence of the checks’ precision on your own workload. Synthetic tests like ours don’t establish it. Confirm an ambiguous alarm A deep health check is the main tool for ambiguous alarms. Its stress tests catch faults that the hardware reports, and subtle data-dependent faults often pass. To run one on the suspect instance group: aws sagemaker start-cluster-health-check \ –cluster-name \ –deep-health-check-configurations ‘[{“InstanceGroupName”:” “,”InstanceIds”:[“i-0123456789abcdef0″],”DeepHealthChecks”:[“InstanceStress”]}]’ Replace the node When you’re ready to replace the node, you have two options. Both trigger the same SageMaker HyperPod recovery process, described in Manually quarantine, replace, or reboot a node . If automatic node recovery is turned on, you can label the node and let SageMaker HyperPod replace it, which is convenient for automation: kubectl label node \ sagemaker.amazonaws.com/node-health-status=UnschedulablePendingReplacement –overwrite To use the API instead, run: aws sagemaker batch-replace-cluster-nodes –cluster-name –node-ids On clusters that use Karpenter for autoscaling, BatchReplaceClusterNodes terminates the node but creates a replacement only if pending pods need one. After the replacement joins, the training job restarts through the SageMaker HyperPod training operator or the job’s restart policy, and your training script resumes from the newest checkpoint saved before the alarm’s suspect_window starts. Call find_divergent_ranks right after loading it, before the first step. Replacement destroys the node’s root and secondary volumes, so keep checkpoints on shared storage. Automating the workflow The manual steps are for understanding. In production, wrap them in a small controller that runs in your own cluster and account: Trigger: The training job reports the alarm as a Kubernetes event, a small custom resource (for example SDCReport{rank, job, step, check}), or an Amazon CloudWatch alarm on metrics such as sdc_nonfinite_grad , sdc_replay_mismatch , and sdc_divergent_tensors . Resolve: The controller maps rank → pod → node through the Kubernetes API, then node → instance ID by stripping the hyperpod- prefix. Contain: It cordons the node. Decide: It applies the table in Step 4. The ring audit has already confirmed ambiguous pairs in place, so the alarm carries the verdicts. For an ambiguous result, it runs one node-scoped deep health check. Recover: For a replacement, it asks a human to approve, then calls BatchReplaceClusterNodes , or sets the node-health label if automatic node recovery is turned on. Then it relies on the training operator and your checkpoint reload, and records every step in CloudWatch. Build it as a lightweight Kubernetes operator, or as an Amazon EventBridge rule on the CloudWatch alarm that invokes an AWS Lambda function. Use case: Testing the checks on SageMaker HyperPod We ran the tests twice on one SageMaker HyperPod cluster orchestrated by Amazon EKS, using 4 and then 3 ml.p5.48xlarge instances with 8 NVIDIA H100 80GB HBM3 GPUs each, and PyTorch 2.9.0 (CUDA 13.0) from the AWS Deep Learning Containers training image. The model was a 953-million-parameter Llama-style decoder with 16 layers, a hidden size of 2,048, 16 attention heads, a 32,000-token vocabulary, and 2,048-token sequences. Each GPU processed 2 sequences per step (131,072 tokens per step on 32 GPUs), with fp32 weights, gradients, and optimizer state, and bf16 autocast for compute. We trained with AdamW at a learning rate of 3 × 10⁻⁴ after a 100-step linear warm-up, with betas of (0.9, 0.95), weight decay of 0.1, and gradient clipping at 1.0. The data was synthetic: a copy task with Zipf-distributed tokens whose first half repeats, seeded per rank and step so that any micro-batch can be rebuilt. Fault scenarios Faults were injected in one layer, the MLP down-projection of the eighth of 16 decoder blocks, on a chosen rank and step: Forward faults change the layer’s output but keep its gradient path, like a wrong multiply in the forward pass. Backward-only faults change the gradient of the layer’s input or weight. Weight faults change one GPU’s copy of the stored weight right after the optimizer step: a bit flip, a swap of two values, or a compensating change to two values. A persistent fault recurs in everything the faulty GPU computes, including the replays it runs for other ranks. A transient fault fires once. Every scenario restarts from a snapshot taken after 400 warm-up steps, and step numbers count from it. Unless noted, audits ran every 10 steps with deterministic kernels, weight checks every 10 steps, and a checkpoint every 20. The table shows the 4-instance results. On 3 instances, every scenario raised the same alarm, at the same step, from the same check, naming the same ranks. Only the peer ranks changed. Injected fault Result Caught by NaN in one element of rank 13’s forward output, on step 25 only (not an audit step) Fired on step 25 and named rank 13. The ring replay also ran. Confirmation by rank 21 and rank 13: transient on rank 13 . Step 25 wasn’t applied. Suspect window: steps 21–25 Check 1 One element of rank 13’s forward output ×1,000, on every step from step 23 Steps 23–29 passed check 1 and were applied. The audit at step 30 mismatched (13, 5) and (21, 13) and named rank 13. Step 30 wasn’t applied. Suspect window: steps 21–30 Check 2 The same ×1,000 on audit step 30 only One pair mismatched, (13, 5). Confirmation by rank 21, on a third node, and by rank 13: transient on rank 13 Check 2 The persistent ×1,000 on rank 5, which replays rank 13’s micro-batch Pairs (13, 5) and (5, 29) mismatched. Named rank 5. A single replay of rank 13 by rank 5 would have blamed the healthy rank 13 Check 2 Other forward faults on rank 13, every step from step 23: one element ×10,000, 10% multiplicative noise on the whole output, bit 3 flipped in one bf16 element, and one output column ×1.1 Each named rank 13 at step 30. The bit flip and the ×1.1 column were also caught with default-kernel audits Check 2 One element of the gradient flowing into rank 13’s layer, or of its weight gradient, ×1,000, every step from step 23 (backward only) Each named rank 13 at step 30, from the gradient fingerprints alone Check 2 The weight-gradient fault above, with default-kernel audits No alarm in 50 steps. The checkpoints at steps 20 and 40 were tagged with passing audits and weight checks, because the loss was unaffected and every copy had applied the same corrupted updates Not caught The persistent forward ×1,000 on ranks 13 and 22 at once Pairs (13, 5), (21, 13), (22, 14) and (30, 22) mismatched. Named ranks 13 and 22 Check 2 Bit 22 flipped in one fp32 weight element on rank 0 after the step-31 update, with a checkpoint every 5 steps The forced weight check at checkpoint step 35 named rank 0, and the save didn’t happen. Suspect window: steps 31–35 Check 3 The same, with check_on_checkpoint=False The save at step 35 went ahead from rank 0’s corrupted copy. Its tags showed the last passing weight check at step 30. The ring audit named rank 0 at step 40, from pairs (8, 0) and (0, 24) Check 2 Two values swapped, or +4,096 and −4,096 added to the bit patterns of two values, in one weight tensor on rank 0 after the step-31 update, with no audits and weight checks every 10 steps Each named rank 0 at the step-40 weight check, although the tensor’s plain bit sum hadn’t changed Check 3 Bit 22 flipped in one attention weight on rank 7, after restoring the snapshot (a stand-in for loading a checkpoint) find_divergent_ranks , run before the first step, named rank 7 Check 3 Fault-free runs To measure false alarms, we trained without faults on both cluster sizes, with three schedules each. Every schedule ran the weight check every 10 steps and saved no checkpoints, and each run’s length was fixed in minutes, so the two schedules that audit every step contribute most of the audit steps: Instances Audits Minutes Steps Audit steps Weight checks Alarms 4 Deterministic, every step 45 4,600 4,600 460 0 4 Default kernels, every step 22.5 5,400 5,400 540 0 4 Deterministic, every 10 steps 45 12,800 1,280 1,280 0 3 Deterministic, every step 15 1,600 1,600 160 0 3 Default kernels, every step 7.5 2,000 2,000 200 0 3 Deterministic, every 10 steps 15 4,400 440 440 0 Total 150 30,800 15,320 3,080 0 With 0 alarms in 15,320 audit steps, the rule of three puts a 95 percent upper bound of 3 / 15,320, about 1 in 5,100, on the false-alarm rate per audit step. The NaN/Inf checks didn’t fire on any fault-free step either. The gradient-norm analysis in Why not compare local gradient norms? uses the 4-instance runs only. Beyond DDP The weight comparison isn’t limited to DDP. find_divergent_ranks takes any data-parallel group, so it also applies to tensor, pipeline, and context parallelism, HSDP, and mixture of experts, which are all layouts that keep independent copies of each weight. It can’t help under ZeRO-1 or full FSDP sharding, where each shard’s update is computed on only one rank. We tested the checks and the training loop in this post only with DDP. To use it in another layout, call find_divergent_ranks with the group that holds copies of the same shard. With only two copies, it can detect a disagreement but can’t name the faulty copy. If you’d rather have the service check your nodes for you, SageMaker HyperPod also offers the MultiNodeTraining deep health check. It runs a short, fixed distributed training job across a batch of GPU nodes, with no code changes, and replaces nodes whose training loss turns NaN or Inf, diverges, or doesn’t finish in time. Select it alongside InstanceStress and InstanceConnectivity when you create an instance group, or run it on demand with StartClusterHealthCheck . It complements the checks in this post: use it to screen new and replacement nodes before your job lands on them, and the in-job checks to catch wrong values in your own training that never become NaN or Inf. For details, visit SageMaker HyperPod Deep Health Checks . Clean up To avoid ongoing charges, delete what you created for testing. If you created a HyperPod cluster, delete it with aws sagemaker delete-cluster –cluster-name . That removes only the HyperPod resources. Delete the EKS cluster, its add-ons, and any Amazon FSx volumes separately, after the HyperPod cluster. See Deleting a SageMaker HyperPod cluster . Conclusion In this post, we showed how to detect silent data corruption in data-parallel training and how to hand a suspect node to SageMaker HyperPod for replacement. Three checks cover the three places a fault can land: a NaN/Inf check on each rank’s gradient and on the reduced gradient, a ring replay that compares every GPU’s work with a GPU on another node, and a weight fingerprint comparison after the update. Run the first two before the optimizer step, don’t apply a step that failed a check, and compare weights before every checkpoint. Node names on SageMaker HyperPod EKS clusters turn the suspect rank into an instance ID. Then quarantine the node, roll back to a checkpoint from before the suspect window, and let the node-health label or the Replace API complete the recovery. To get started, review the SageMaker HyperPod resiliency features for Amazon EKS , and try the companion repository . To go deeper, see deep health checks and the observability add-on . About the authors Siddharth Senger Siddharth currently serves as a Senior Software Development Engineer at Amazon Web Services (AWS), specifically within the SageMaker HyperPod team. Bringing nearly a decade of software development experience, Siddharth has contributed to several teams across Amazon, including Retail, Amazon Rekognition, Amazon Textract, and Amazon SageMaker. He is passionate about building reliable, scalable, and efficient distributed systems that empower customers to accelerate large-scale machine learning and AI innovation. Roy Allela Roy is a Principal AI/ML Specialist Solutions Architect at AWS. He works with AWS customers, from startups to large enterprises to train and deploy foundation models efficiently on AWS. He has a background in Microprocessor Engineering. He is passionate about computational optimization problems and improving the performance of AI workloads. Nilesh PS Nilesh is a Senior Software Development Engineer at AWS working on Amazon SageMaker HyperPod. He focuses on Ray cluster management, training resiliency, and observability for large-scale distributed ML workloads on Kubernetes. Pradeep Cruz Pradeep is a Senior Software Development Manager at AWS, leading SageMaker HyperPod infrastructure for enterprise AI, Training, Inference and Model Customization. With over 20 years of experience, he has scaled critical 0-to-1 AI/ML platforms across AWS, T-Mobile, and Ericsson. An expert in distributed systems and infrastructure, he holds an MBA from Michigan Ross.