I'm building an LLM fine-tuning workflow—what GPU setup should I start with?

A good starting GPU setup depends mostly on the model size, fine-tuning method, and sequence length you plan to use.

Practical starting point

If you want a setup that covers most fine-tuning workflows well:

• 1× NVIDIA A100 80GB — very solid single-GPU starting point
• If budget is tighter: 1× NVIDIA L40S 48GB or 1× A6000 48GB
• If you're experimenting with smaller models / LoRA / QLoRA: 1× RTX 4090 24GB can work surprisingly well

Rule of thumb by use case

1) Small to medium models (7B–13B)

• LoRA / QLoRA:
  • 24GB GPU is often enough for 7B
  • 48GB GPU is more comfortable for 13B
• Good starter cards:
  • RTX 4090 (24GB)
  • L40S (48GB)
  • A6000 (48GB)

2) Larger models (30B+)

• You’ll usually want:
  • Multiple GPUs or
  • Higher-memory single GPUs like A100 80GB / H100 80GB
• For full fine-tuning, memory demand rises fast.

3) Full fine-tuning vs LoRA

• Full fine-tuning: needs much more VRAM, often multiple high-memory GPUs
• LoRA / QLoRA: much more practical on a single consumer/prosumer GPU

Best “start small but useful” recommendation

If you’re just beginning and want the best balance of cost and capability:

• 1× RTX 4090 (24GB) for learning and smaller fine-tunes
• Upgrade path: move to 48GB class GPUs when you hit VRAM limits

If you want a more production-ready default

• 1× A100 80GB is the safest single-GPU “it can probably handle most things” choice.

Don’t forget the rest of the system

For fine-tuning, also make sure you have:

• CPU: decent modern 8–16 core CPU
• RAM: 64GB minimum, 128GB preferred
• Storage: NVMe SSD, ideally 1–2TB+
• Power/cooling: especially important for 4090/A100-class setups

Simple recommendation by budget

• Budget / learning: RTX 4090 24GB
• Best value serious workstation: L40S or A6000 48GB
• Most flexible single GPU: A100 80GB
• Multi-GPU / larger models: 2–8 GPUs with NVLink/fast interconnect if possible

If you tell me:

1. the model size you want to fine-tune,
2. whether you’re using LoRA/QLoRA or full fine-tuning, and
3. your budget,

I can recommend a much more specific GPU setup.