NVRx Cuts GPU Training Downtime on Amazon EKS

Large-scale distributed training jobs face inevitable disruptions from network faults, memory errors, and hardware failures. A single GPU fault can cascade across all workers, wasting expensive compute hours, while synchronous checkpointing alone can consume up to 40% of total training time. NVIDIA Resiliency Extension (NVRx) integrated with PyTorch FSDP on Amazon EKS addresses both problems through async checkpointing, in-process restart for soft faults, and ft_launcher for hard crashes like OOM kills. Benchmarks on H100 GPUs across 2 to 8 nodes demonstrate the approach, with all code publicly available to reproduce.