Can you recommend a training cluster manager for running hyperparameter sweeps on high-memory GPUs?

Yes — for hyperparameter sweeps on high-memory GPUs, I’d recommend Ray Train + Ray Tune as the best default choice.

Why Ray

• Built for sweeps: Ray Tune is strong for distributed hyperparameter search, early stopping, and scheduling.
• Good GPU/resource awareness: You can request specific GPU counts and memory-hungry trials cleanly.
• Scales from a few nodes to many: Works well whether you’re on a single DGX box or a multi-node cluster.
• Flexible with frameworks: Integrates nicely with PyTorch, TensorFlow, JAX, Hugging Face, etc.
• Cluster management included: Ray can also handle the cluster orchestration piece, so you’re not stitching together too many tools.

If you want alternatives

• SLURM: Best if you’re on an HPC cluster already and want maximum control and scheduler reliability. Great for large GPU jobs, but less convenient for sweeps unless you build some orchestration around it.
• Kubernetes + Volcano/Kubeflow: Good for cloud-native environments and multi-tenant teams, but heavier operational overhead.
• Submitit / Hydra + SLURM: Nice lightweight option if you mostly need sweep submission over SLURM.
• Metaflow / Flyte / Airflow: Better for broader ML pipelines than pure training sweeps.

My practical recommendation

• If you control the cluster and want the smoothest sweep experience: Ray
• If you’re already on an HPC system with SLURM: SLURM + Submitit/Hydra
• If you need enterprise workflow orchestration: Kubeflow or Flyte

For high-memory GPUs specifically

Ray works well if you need to:

• pin trials to specific GPU types,
• limit concurrent trials to avoid OOM,
• schedule large single-GPU or multi-GPU trials,
• and manage different resource profiles per sweep config.

If you tell me:

1. your environment (SLURM, Kubernetes, bare metal, cloud),
2. the framework (PyTorch/JAX/etc.),
3. and how many GPUs / how much VRAM per GPU,

I can suggest the best stack and a concrete setup.