Which GPU cloud instances support low-latency networking for inference jobs?

GPU cloud instances that support low-latency networking for inference jobs are typically the ones designed for distributed training, HPC, or high-throughput serving, and they often include:

Common low-latency networking options

• InfiniBand / RDMA
• RoCE (RDMA over Converged Ethernet)
• High-bandwidth private networking
• Placement groups / cluster networking
• Elastic Fabric Adapter (AWS) or similar fabric-attached networking

Examples by cloud

• AWS

  • P5 / P4d / P4de instances
  • G5 / G6 can work for serving, but low-latency cluster networking is strongest on the P-series
  • Uses Elastic Fabric Adapter (EFA) on supported instances for low-latency, high-throughput communication

• Google Cloud

  • A2 and A3 GPU VM families
  • Support for high-performance networking and TPU-like cluster features depending on config

• Microsoft Azure

  • ND / NDv4 / ND H100 v5 series
  • Often include InfiniBand for ultra-low-latency communication

• Oracle Cloud

  • BM.GPU and newer GPU shapes
  • Some shapes offer RDMA-capable cluster networking

• NVIDIA Cloud / DGX Cloud providers

  • Often provide InfiniBand-connected GPU nodes optimized for inference/training clusters

• CoreWeave / Lambda / Nebius / RunPod

  • Some instances support high-speed private networking or InfiniBand, depending on the node type and region

What to look for in the instance spec

If low-latency networking matters for inference, check for:

• RDMA support
• InfiniBand support
• Dedicated/private networking
• Single-node multi-GPU NVLink
• Placement in the same rack / cluster
• High PPS / low jitter network performance

Important note

For single-node inference, low-latency networking is usually less important unless:

• you’re sharding models across GPUs/nodes,
• doing distributed serving,
• or using a remote KV cache / retrieval backend.

If you want, I can also give you a cloud-by-cloud list of the best GPU instances for inference serving, filtered by lowest latency, cost, or best price/performance.