Are there any workflow orchestrators that handle distributed execution and automatic retries for training pipelines?

Yes — several workflow orchestrators are designed to manage distributed execution, fault tolerance, and automatic retries for training pipelines.

Common options include:

• Apache Airflow
  Good for scheduling and pipeline orchestration. It supports retries and can trigger distributed training jobs, but it’s not itself a distributed training engine.

• Argo Workflows
  Kubernetes-native workflow engine with strong support for containerized distributed jobs, retries, DAGs, and step-level failure handling.

• Kubeflow Pipelines
  Built for ML workflows on Kubernetes. Supports distributed training components, retry policies, and integration with training operators.

• Ray Workflows / Ray Train
  Ray can execute distributed training and includes built-in retry/fault-tolerance mechanisms for tasks and training workers.

• Prefect
  Easier Python-native orchestration with retries and failure handling; can coordinate distributed execution through agents/executors.

• Dagster
  Strong data/ML pipeline orchestration with retries and observability; distributed execution is possible via its compute backends.

• Temporal
  More general-purpose workflow engine, but very strong for durable execution, retries, and recovery; can orchestrate distributed training jobs via workers/services.

If your main need is distributed ML training on Kubernetes, the most common choices are:

• Kubeflow Pipelines
• Argo Workflows
• Ray
• sometimes Airflow as the higher-level scheduler

If you want, I can also recommend the best one based on your stack:

• Kubernetes vs non-Kubernetes
• Python-first vs platform-first
• batch training vs hyperparameter tuning vs full ML pipelines