How do I estimate how many GPUs I need for fine-tuning?

A practical way to estimate GPU needs for fine-tuning is to work backward from:

1. Model size
2. Sequence length
3. Batch size / gradient accumulation
4. Precision and optimizer
5. Whether you’re doing full fine-tuning, LoRA/QLoRA, or freezing most weights

1) Start with the type of fine-tuning

Full fine-tuning

You update all model weights. This is the most memory-hungry.

LoRA / QLoRA

You update only small adapter weights, so memory needs drop a lot. This can often turn “needs many GPUs” into “fits on 1 GPU.”

2) Estimate memory per GPU

A rough memory breakdown:

• Model weights
• Gradients
• Optimizer states (often the biggest part in full fine-tuning)
• Activations during forward/backward pass
• Temporary overhead from framework/CUDA

Rule of thumb for full fine-tuning

With Adam optimizer in mixed precision, training memory can be roughly:

• Weights: ~2 bytes/parameter (fp16/bf16) or 4 bytes if stored in fp32
• Gradients: ~2 bytes/parameter
• Optimizer states: often ~8–12 bytes/parameter total
• Total: about 12–20 bytes per parameter, sometimes more with overhead

So:

• 7B params → roughly 84–140 GB total training memory
• 13B params → roughly 156–260 GB
• 70B params → roughly 840 GB–1.4 TB

This is why full fine-tuning large models usually requires multiple GPUs or sharding.

3) Divide by GPU memory, but leave headroom

Example:

• Using 80 GB A100s
• A 13B full fine-tune might need ~200 GB total

So you’d need at least:

• 200 / 80 = 2.5, so realistically 3 GPUs minimum
• In practice, 4 GPUs is safer because activations and overhead can spike

4) Batch size and sequence length matter a lot

Activation memory scales strongly with:

• batch size
• sequence length
• model depth/hidden size

If you increase sequence length from 2k to 8k, memory can rise dramatically.

To reduce GPU count:

• lower per-GPU batch size
• use gradient accumulation
• use shorter sequences
• enable gradient checkpointing
• use flash attention if supported

5) Use these quick heuristics

Full fine-tuning

• 7B model: often 1–2× 80GB GPUs with good memory optimization
• 13B model: often 2–4× 80GB GPUs
• 70B model: typically 8+× 80GB GPUs

LoRA / QLoRA

• 7B model: often 1× 24GB or 1× 48GB
• 13B model: often 1× 48GB or 1× 80GB
• 70B model: may still be possible on a few GPUs with quantization, but depends heavily on sequence length and implementation

6) Compute is different from memory

Even if the model fits on one GPU, training may be too slow. For time estimates, you need to consider:

• dataset size
• number of epochs
• tokens per step
• GPU FLOPs

A single GPU may be enough for experimentation, but more GPUs reduce wall-clock time.

7) Best way to estimate in practice

1. Pick your fine-tuning method: full FT vs LoRA/QLoRA
2. Estimate max sequence length
3. Estimate per-GPU batch size
4. Run a small test with:
   • same model
   • same sequence length
   • same precision
   • same optimizer
5. Watch peak memory usage
6. Scale up with a 20–30% safety margin

8) Simple formula to approximate

For full fine-tuning:

Total memory needed ≈ parameter_memory + optimizer_memory + activation_memory + overhead

Then:

# GPUs ≈ total_memory_needed / usable_memory_per_GPU

Where usable memory is usually only 70–85% of advertised VRAM.

---

If you want, I can help you estimate GPU count for a specific setup if you tell me:

• model name/size
• sequence length
• batch size
• full fine-tune or LoRA/QLoRA
• GPU type (e.g. 24GB, 48GB, 80GB)