By: Alex Mercer – SeaPRwire – AI training teams lose hours every week to the same old problem. GPU clusters fail. Work restarts. Progress vanishes. Clockwork.io just drew a hard line against this waste. They launched the YOCO Guarantee. It promises at least 90 percent of training failures on supported TorchPass workloads get fixed through live GPU migration. No lost progress. No checkpoint rollback. No recompute. Miss the mark in any contract year and customers get a 25 percent credit on their next TorchPass renewal or expansion. This shifts the conversation. It moves beyond old uptime metrics. It focuses on what actually matters. Does the job finish on time.
The numbers expose the pain. Research from Meta FAIR at HPCA 2025 shows a 1,024-GPU cluster has a mean time to failure of just 7.9 hours. Scale to 16,384 GPUs and it drops to 1.8 hours. Each failure triggers node replacement, checkpoint restore, and full recompute of every step since the last save. That cycle eats three or more hours of progress per event. Losses stack up daily. Typical GPU clusters run at only 30 to 50 percent of theoretical performance. The hardware is capable. The reliability model is not. In a 2,048-GPU H200 setup the annual waste exceeds six million dollars. That covers idle recovery time, cascading retries, and recomputed steps. Suresh Vasudevan, CEO of Clockwork.io, put it plainly. AI teams need models done, not nodes up. Most contracts guarantee node availability. They ignore job continuity. The result feels unreliable to operators even when SLAs get met on paper. Recompute is the hidden tax. Many teams accept it as normal. Clockwork.io says it does not have to be.
TorchPass changes the mechanics. It makes reliability software-defined. Live GPU migration moves the full in-memory state. Model weights, gradients, optimizer state all transfer to a healthy node. Training picks up exactly where it left off. Recovery usually takes about three minutes. No restore. No recompute. The system handles three failure types. Unplanned migration covers sudden crashes, power loss, or GPU faults using healthy replicas. Pre-emptive migration acts on early signals like rising ECC errors or thermal issues. Planned migration supports maintenance, patching, and updates without stopping work. Across all cases the job keeps running. This cuts wasted training progress by 90 percent. Lost time in a 1,024-GPU cluster falls from roughly three hours per day to under ten minutes. Research teams avoid silent erasures of progress. Model timelines turn predictable. Independent testing by SemiAnalysis confirmed TorchPass outperforms other fault-tolerance options. It is the only solution that keeps the same training performance as jobs without fault tolerance. It works in cloud and on-premises. It supports TorchTitan, Megatron-LM, DeepSpeed. Schedulers include Kubernetes and Slurm. It runs on NVIDIA and AMD hardware across InfiniBand, RoCE, and Ethernet. No hardware lock-in. Jordan Nanos from SemiAnalysis noted the results in testing. TorchPass delivered the fastest fault-tolerant performance for a GPT-OSS-120B run on a 64x H200 cluster. It beat checkpoint-restart on completion time. It outperformed TorchFT on MFU and tokens per second per GPU while matching recovery time. The guarantee simply puts that performance into the contract.
Fred Bardolle, Head of Products and AI at Scaleway, highlighted the shift. Every enterprise knows the cost of a failed job. Hours lost. Recomputes billed. Timelines slip. Product decisions at Scaleway center on predictable outcomes. Node uptime answers the wrong question. The YOCO Guarantee targets the right metric. Progress stays protected. Jobs run to completion. The guarantee becomes available to new and renewing customers on August 3, 2026. Existing customers can contact their account team. Clockwork.io will discuss the details at RAISE Summit in Paris on July 8-9. Vasudevan joins a panel on infrastructure. The move forces a broader market rethink. AI builders now have a clear SLA question. What percentage of training failures resolve without lost progress. This metric ties to GPU ROI. Operators gain a competitive edge with contractual job continuity. They reduce idle time and command better pricing. Vendors without similar backing compete mainly on raw GPU cost. The industry gains a measurable standard. Vendor claims now face contractual teeth. Clockwork.io puts skin in the game. They built TorchPass on Software-Driven AI Fabrics. It delivers telemetry, fault tolerance, and optimization. Customers like Uber, Wells Fargo, DCAI, Nebius, NScale, and White Fiber already rely on it. The guarantee turns testing results into enforceable commitments.
AI infrastructure contracts have treated failure recovery as optional. Clockwork.io makes it mandatory and measurable. Teams evaluating new setups should demand similar accountability. Ask for credits tied to job completion rates. Test live migration under real workloads. Track actual progress lost rather than node uptime alone. Contracts built around the right metric cut waste fast. Start there and the numbers improve quickly.
Author bio: Alex Mercer, senior commentator for an international frontline tech weekly with over 15 years covering enterprise software and industry transformation.