What’s the right GPU profile for a 70B model?

For a 70B parameter model, the “right” GPU profile depends mainly on precision, context length, and whether you’re doing inference vs. training/fine-tuning.

Quick rule of thumb

Inference

• FP16/BF16: needs roughly 140 GB just for weights
• INT8: roughly 70–80 GB
• 4-bit quantized: roughly 35–45 GB

Practical GPU profiles

• Single H100 80GB / A100 80GB: can run 70B only with quantization or aggressive memory optimizations
• 2×80GB GPUs: comfortable for BF16/FP16 inference
• 4×48GB or 4×40GB GPUs: common for BF16/FP16 inference with tensor parallelism
• 1×24GB GPU: only feasible with 4-bit quantization, reduced context, and careful offloading

If you mean hosting the model for inference

A good starting point is:

• 2×80GB GPUs for easy BF16
• 1×80GB GPU for 4-bit or 8-bit quantized deployment
• 4×A10/L40S/4090-class GPUs if you’re okay with lower precision and distributed inference

If you mean training or fine-tuning

• Full fine-tuning of 70B is extremely expensive and usually needs many high-memory GPUs
• LoRA / QLoRA can make it possible on:
  • 1×48GB
  • 1×80GB
  • or a small multi-GPU setup

Short answer

If you want a safe recommendation for running a 70B model well, aim for:

• 80GB GPU minimum
• 2×80GB preferred for BF16/FP16
• 4-bit quantization if you only have 24–48GB

If you want, I can give you a specific GPU recommendation based on:

1. inference or training
2. context length
3. precision/quantization
4. your exact GPU options/budget