Can you recommend a checkpoint storage service for resuming distributed training runs after interruptions?

Yes — if your goal is to resume distributed training after interruptions, the checkpoint store should be:

• durable and highly available
• fast enough for large checkpoint files
• accessible from multiple workers / nodes
• ideally supports versioning and lifecycle management

Good options by setup

1) Amazon S3 — best general-purpose choice

• Very common for training pipelines
• Durable, scalable, easy to use from most frameworks
• Works well with multi-node training if all workers can read/write the same bucket
• Pairs nicely with versioning and lifecycle policies

Best for: AWS-based or cloud-agnostic distributed training
Tradeoff: Not the lowest-latency storage for very frequent checkpoint writes

2) Google Cloud Storage (GCS) / Azure Blob Storage

• Similar role to S3 on their respective clouds
• Strong durability and easy integration with ML stacks

Best for: GCP/Azure environments

3) Shared file systems: EFS / FSx for Lustre / NFS / Lustre

• Useful when checkpointing very frequently and needing low-latency access
• Works well for jobs that expect POSIX filesystem semantics

Best for: HPC-style or tightly coupled training clusters
Tradeoff: More operational complexity, and not always ideal for long-term durable storage unless backed up elsewhere

4) Managed ML checkpointing services

Depending on your platform:

• AWS SageMaker checkpoints
• Vertex AI / Azure ML equivalents
• These can simplify integration and recovery

Best for: Teams already using a managed ML platform

---

My practical recommendation

For most distributed training setups:

• Use object storage (S3/GCS/Blob) as the durable checkpoint store
• Optionally write intermediate checkpoints to a shared low-latency filesystem
• Keep the “latest” checkpoint plus a few older versions
• Save:
  • model weights
  • optimizer state
  • scheduler state
  • RNG states
  • training step / epoch metadata

---

If you want a single recommendation:

Amazon S3 is usually the safest default for resilient distributed-training checkpoint storage.

If you tell me your environment (AWS/GCP/Azure/on-prem, framework like PyTorch Lightning, DeepSpeed, Ray, Horovod, etc.), I can recommend a more specific setup.