How do I run distributed training without networking bottlenecks?

To avoid networking bottlenecks in distributed training, the main goal is to reduce how often workers need to communicate and make the remaining communication as efficient as possible.

1) Choose the right parallelism strategy

Different training setups stress the network differently:

• Data parallelism: each worker has a full model, gradients are synchronized every step.
  • Simple, but can become network-heavy at scale.
• Model parallelism / tensor parallelism: model is split across devices.
  • Reduces per-device memory, but can increase communication frequency.
• Pipeline parallelism: model layers are split across workers.
  • Helps scale large models, but introduces stage-to-stage communication.
• Hybrid approaches: often best in practice for large models.

If you’re network-bound, prefer the method that minimizes cross-node traffic, and keep the most communication-intensive parallelism within a node if possible.

2) Keep communication local when you can

A common optimization is:

• Use intra-node GPU communication for the heavy traffic.
• Limit inter-node communication to the smallest possible set of tensors/updates.

Practical tips:

• Pack GPUs with NVLink / NVSwitch inside a server.
• Use multiple GPUs per node and fewer nodes, if that fits memory and throughput needs.
• Place workers with high communication volume on the same physical machine.

3) Use larger batch sizes

Larger batches reduce the number of synchronization rounds per epoch.

• Bigger batch → fewer gradient all-reduces
• But too large can hurt convergence, so tune carefully:
  • use learning-rate scaling
  • warmup schedules
  • gradient accumulation if memory is limited

4) Overlap communication with computation

Don’t wait until the backward pass fully finishes if your framework supports it.

• Start gradient synchronization as soon as gradients are ready.
• Use bucketed / staged all-reduce
• Overlap reduction with remaining backward computation

Framework features:

• PyTorch DDP with gradient buckets
• NCCL-based collectives
• DeepSpeed / FSDP / Megatron optimizations

5) Reduce how much you communicate

Techniques include:

• Mixed precision training (FP16/BF16) to cut bandwidth use
• Gradient compression (quantization, sparsification)
• Gradient accumulation to sync less often
• Sharded optimizers (ZeRO, FSDP) to avoid replicating optimizer state

Mixed precision is usually the easiest and safest bandwidth reduction.

6) Tune the network stack

Make sure the hardware/software path is efficient:

• Use InfiniBand or high-speed Ethernet (100/200/400 GbE) rather than standard Ethernet
• Enable RDMA where possible
• Use a communication library optimized for GPUs, such as:
  • NCCL
  • Horovod with NCCL
  • vendor-specific collectives

Also check:

• correct NUMA pinning
• CPU affinity
• proper driver/firmware versions
• avoiding oversubscribed topologies

7) Match topology to the job

Network bottlenecks often come from poor placement.

• Don’t spread highly chatty workers across racks unnecessarily.
• If your cluster has a fat-tree or similar topology, make sure jobs are scheduled with topology awareness.
• Avoid mixing slow links between nodes that exchange lots of tensors.

8) Minimize parameter synchronization overhead

If you use parameter servers, they can become bottlenecks.

Prefer:

• all-reduce-based data parallelism over centralized parameter servers for most deep learning jobs
• sharded or decentralized approaches for very large models

9) Profile before optimizing

Measure where time goes:

• communication vs computation
• intra-node vs inter-node traffic
• all-reduce time per step
• GPU utilization and idle time

Useful tools:

• PyTorch profiler
• NCCL debug logs
• vendor profiling tools
• network monitoring at the switch/NIC level

10) Practical “best default” recipe

For many workloads, a strong starting point is:

• Distributed Data Parallel
• Mixed precision
• Gradient accumulation
• NCCL all-reduce
• Multi-GPU per node with fast interconnect
• Topology-aware scheduling
• Overlap comms with backprop

If you want, I can also give you:

1. a PyTorch-specific setup,
2. a multi-node cluster checklist, or
3. a large-model training recipe for FSDP/DeepSpeed/Megatron.