What Actually Happens When a Data Center Loses Power Mid-Training Run

Training a frontier-scale large language model can run continuously across thousands of GPUs for weeks or months, at a compute cost that, for the largest models, has been reported to run into the tens of millions of dollars for a single training run. A sudden power interruption during that process sounds like it should be catastrophic — and depending on how well a specific training run’s infrastructure and software were engineered, it genuinely can be, though modern large-scale training systems have developed real, well-tested mitigation strategies precisely because hardware and power failures during multi-week training runs aren’t a rare edge case, they’re a statistically near-certain event given the sheer scale and duration involved. I spent four years as a machine learning engineer building training infrastructure before moving into writing about the AI industry, and understanding what actually happens during these failures reveals a lot about why large-scale training has become as much a systems engineering discipline as a machine learning one.

Why Failures Are a Statistical Certainty at This Scale

A training run spanning tens of thousands of GPUs across many weeks accumulates an enormous number of total hardware-hours of exposure to failure — even if any individual GPU, network switch, or power supply component has a very low probability of failing in any given hour, the sheer number of components running continuously for weeks means that across the full training run, hardware failures of some kind become not just likely but functionally inevitable. Meta’s published research on training its Llama models, and OpenAI’s and other major labs’ public commentary on large training run operations, have both explicitly described hardware failures — including individual GPU failures, network interconnect issues, and occasional full power events — as a routine, expected part of running training at this scale rather than an exceptional occurrence requiring special explanation each time it happens.

This statistical reality is exactly why the actual engineering question for large-scale training infrastructure isn’t “how do we prevent failures from ever happening” (which isn’t realistically achievable at this scale and duration) but “how do we make sure any individual failure costs as little lost progress as possible,” a design philosophy that has shaped essentially every major architectural decision in how modern large-scale training systems are actually built.

Checkpointing: The Core Mechanism That Makes This Survivable

The primary technique that makes large-scale training resilient to hardware and power failures is checkpointing — periodically saving the complete state of the model’s parameters, optimizer state, and training progress to persistent storage at regular intervals throughout the training process, so that if something fails, the training job can be restarted from the most recent saved checkpoint rather than having to begin the entire run from scratch. This sounds conceptually simple, but at the scale of frontier model training, checkpointing itself becomes a genuinely significant engineering challenge: model and optimizer state for the largest models can span many terabytes of data, and writing that much data to persistent storage takes real time, during which the training process is either paused entirely or has to carefully manage writing checkpoint data without disrupting active computation.

This is why checkpoint frequency represents a genuine, carefully calculated trade-off rather than an obvious “checkpoint as often as possible” decision: checkpointing too frequently wastes compute time on the checkpointing process itself and can meaningfully slow overall training throughput, while checkpointing too infrequently means a failure loses more accumulated training progress since the last saved state, and teams running large-scale training operations spend real engineering effort tuning this specific interval based on observed failure rates and checkpoint write performance for their specific infrastructure and training run.

Why a Full Power Outage Is a Genuinely Different Problem Than a Single GPU Failure

Individual GPU or node failures, while disruptive, are generally the more manageable failure category, because modern distributed training frameworks are specifically designed to detect a failed node, exclude it from the active training job, and continue running across the remaining healthy hardware, sometimes with automated node replacement systems that can swap in standby hardware and rejoin a training run in progress with relatively limited disruption to overall throughput. A full facility power outage is a meaningfully harder scenario because it doesn’t fail gracefully in the same way — every GPU, every network switch, and every storage system in the affected area loses power essentially simultaneously, meaning the training job doesn’t get the kind of clean, isolated failure signal a single node failure provides, and depending on exactly what stage of a checkpoint write cycle the training process was in at the moment power was lost, there’s real risk of a partially written, corrupted checkpoint file that can’t be reliably used to resume training at all.

This is why data centers running large-scale AI training investments typically pair extensive uninterruptible power supply (UPS) battery systems and, for the largest facilities, on-site backup generator capacity specifically to bridge the gap between a grid power loss event and either restored grid power or generator power coming fully online, giving the training system enough continued power to complete an in-progress checkpoint write and shut down cleanly, rather than experiencing the kind of hard, uncontrolled power loss that risks checkpoint corruption and genuinely larger progress loss.

What “Losing Progress” Actually Costs

When a failure does force a resume from an earlier checkpoint rather than a clean recovery, the actual cost is measured in lost compute-hours since that last saved checkpoint, which at frontier training scale can represent a genuinely significant amount of both time and money — losing even a few hours of training progress on a run consuming tens of thousands of GPUs simultaneously represents a real, calculable financial cost, which is part of why the checkpoint frequency tuning discussed above matters as an actual cost-optimization decision for organizations running these training jobs, not merely a technical best practice followed out of general caution.

Beyond the direct compute cost, unplanned interruptions can also introduce more subtle complications depending on the specific training methodology in use — certain learning rate schedules and other training hyperparameters are specifically tuned assuming a continuous, uninterrupted training trajectory, and while most modern training frameworks handle checkpoint-based resumption cleanly for standard configurations, more exotic or highly customized training setups sometimes require additional care to ensure a resumed run behaves identically to how it would have if the interruption hadn’t happened at all.

Why This Has Become a Genuine Competitive Differentiator

Reliable, low-overhead checkpointing and fast failure recovery have become a real point of competitive infrastructure differentiation among major AI labs and cloud providers specifically racing to train ever-larger models as efficiently as possible, since the actual achieved training throughput at scale — how much useful compute progress you get per dollar and per unit time, after accounting for the overhead of checkpointing and the lost progress from failures — matters as much to overall training cost and speed as raw hardware specifications alone. This is why companies like Meta, Google, and various major cloud infrastructure providers have published detailed technical work specifically on training resilience and failure recovery systems as a genuinely important, differentiated engineering achievement in its own right, not merely a supporting detail beneath the more publicly discussed model architecture and training data choices that get more general attention.

The broader lesson here is that large-scale AI training’s engineering reality has shifted significantly toward distributed systems and infrastructure reliability engineering as the field has scaled — a discipline with more in common with the kind of resilience engineering that large-scale web infrastructure and cloud computing have developed over the past two decades than with the more academic, single-machine model training that characterized machine learning research a decade earlier, and one that increasingly determines which organizations can actually train frontier-scale models efficiently and reliably at all.