Checkpoints
Periodic saving of model state during training for fault tolerance and early stopping
Intent & Description
🎯 Intent
Handle training interruption and identify the best-performing point in training, which isn’t necessarily the final step, by periodically saving complete model state.
📋 Context
Training a nontrivial model can take a long time and is vulnerable to interruption—crashes, preempted instances, hardware failure. The best-performing point in training might be several epochs earlier, before overfitting set in.
💡 Solution
Periodically save the complete state of a model during training—weights at minimum, ideally optimizer state and training metadata too. This buys fault tolerance (resume after crash), natural implementation of early stopping (pick best checkpoint by validation performance), and flexibility to pause, resume, or fine-tune from any earlier point.
Real-world Use Case
- Long-running training jobs on expensive hardware
- Training where early stopping is needed
- Distributed training prone to failures
- Any non-trivial training run where time/cost matters
Source
📌 TL;DR
Periodically save complete model state during training to enable fault tolerance, early stopping, and the ability to resume or fine-tune from any point
Advantages
- Fault tolerance and recovery from interruptions
- Natural early stopping implementation
- Flexibility to resume or fine-tune from any point
- Ability to select best model by validation performance
Disadvantages
- Storage overhead for multiple checkpoints
- Requires discipline about which checkpoint is the model
- Adds complexity to training infrastructure
- Needs careful management of checkpoint retention