What GPU should I use for training a 13B model?

For training a 13B parameter model, the “right” GPU depends on whether you mean:

1. Full training / pretraining from scratch
2. Fine-tuning
3. QLoRA / LoRA adapter training

Quick answer

• Full training from scratch: use multiple high-memory GPUs, typically 8× A100 80GB or similar, because a 13B model is very memory- and compute-intensive.
• Full fine-tuning: a single 80GB GPU can sometimes work with aggressive optimizations, but 2–4 GPUs with 48–80GB each is much more realistic.
• LoRA / QLoRA fine-tuning: you can often use 1× RTX 4090 (24GB) or 1× A100 40/80GB.

Practical recommendations

Best value for fine-tuning

• RTX 4090 24GB: good for LoRA/QLoRA, cheapest strong option
• RTX 6000 Ada 48GB: much more comfortable
• A100 80GB: best single-GPU experience if budget allows

For serious full fine-tuning

• 2× A100 80GB
• 4× L40S 48GB
• 8× A100 80GB for large batch sizes or longer context

Rough memory needs

A 13B model in:

• FP16/BF16 weights alone: about 26GB
• Training states (gradients + optimizer): can push total requirements into 100GB+ range for full fine-tuning
• With ZeRO, FSDP, gradient checkpointing, offloading, the requirement drops a lot

My simple recommendation

• If you’re fine-tuning: get an RTX 4090 if you want value, or A100 80GB if you want ease and speed.
• If you’re training from scratch: plan on multi-GPU server hardware, not a single consumer GPU.

If you want, I can give you a recommendation based on:

• your budget
• whether you mean pretraining vs fine-tuning
• your target context length
• your framework (PyTorch, DeepSpeed, FSDP, etc.)