How do I use spot instances safely for model training?

Using spot instances for model training can save a lot of money, but you need to assume they can disappear at any time. The key is to make training restartable, checkpointed, and fault-tolerant.

Core practices

1. Save checkpoints frequently

• Write model checkpoints to durable storage (S3, GCS, Azure Blob, etc.).
• Save:
  • model weights
  • optimizer state
  • scheduler state
  • current epoch/step
  • random seeds if needed for exact reproducibility

Rule of thumb: checkpoint every few minutes or every N training steps, not just at epoch boundaries.

2. Design training to resume cleanly

Your training script should:

• detect the latest checkpoint on startup
• load model + optimizer + scheduler state
• continue from the correct global step/epoch
• avoid repeating or skipping data when possible

3. Use idempotent data processing

If a job is interrupted and restarted:

• data loading should be repeatable
• temporary files should be written safely
• outputs should be versioned or atomic
• avoid assuming “exactly once” execution

4. Separate training state from ephemeral compute

Keep important artifacts outside the spot instance:

• checkpoints
• logs
• metrics
• configs
• final model export

The instance itself should be disposable.

5. Make evaluation and exporting robust

• Run evaluation from checkpoints, not only from the live job.
• Export final artifacts only after checkpoint validation.
• Consider a separate post-processing job on on-demand instances.

6. Use distributed training carefully

If using multi-node spot training:

• expect partial node loss
• prefer frameworks that support elastic training or worker replacement
• synchronize checkpointing across all workers
• test failure scenarios before relying on it

7. Monitor interruption signals

Cloud providers often give a warning before termination:

• AWS Spot Instance interruption notice
• GCP preemption notice
• Azure low-priority VM eviction notice

Use that warning to:

• save a checkpoint immediately
• gracefully stop training
• flush logs/metrics

8. Keep jobs short enough to tolerate restarts

Long jobs lose more work when interrupted. Safer approach:

• break training into smaller runs
• checkpoint often
• plan for multiple restarts

Practical architecture

A common safe setup looks like this:

1. Start training on spot instances
2. Every few minutes:
   • checkpoint to object storage
   • upload metrics
3. On interruption notice:
   • save one final checkpoint
   • exit cleanly
4. A controller or launcher:
   • detects failure
   • restarts a new spot job
   • resumes from the latest checkpoint

Extra safeguards

• Use a small amount of on-demand capacity for critical components like:
  • orchestration
  • dataset indexing
  • experiment tracking
• Store a manifest of completed checkpoints.
• Validate checkpoints periodically by loading them in a separate process.
• If data preprocessing is expensive, preprocess once and store results durably.

When spot instances are a bad fit

Avoid or limit spot usage if:

• the training job cannot be restarted safely
• you have very long checkpoint intervals
• training is extremely stateful and hard to resume
• deadlines are tighter than the expected interruption rate
• you cannot tolerate occasional wasted work

Simple checklist

Before using spot instances, confirm:

• training can resume from checkpoint
• checkpoints include optimizer/scheduler state
• checkpoints go to durable storage
• interruption handling is implemented
• logs/metrics are persisted externally
• restart logic has been tested
• data pipeline is restart-safe

If you want, I can also give you:

1. a PyTorch example for spot-safe checkpointing, or
2. a cloud-specific guide for AWS/GCP/Azure spot training.