I'm building a setup for fine-tuning image models—what GPU memory do I need?

It depends a lot on which image model you’re fine-tuning and how you’re fine-tuning it, but here’s a practical rule-of-thumb guide.

Quick answer

• 8 GB VRAM: possible for small models, low batch sizes, LoRA/QLoRA-style tuning, or heavy gradient checkpointing.
• 12–16 GB VRAM: a good entry point for many common fine-tuning jobs.
• 24 GB VRAM: comfortable for a lot of image fine-tuning work.
• 48 GB+ VRAM: best if you want larger batch sizes, higher resolutions, or training larger diffusion/vision models more easily.

By workload

1) Classic CNNs / smaller vision models

Examples: ResNet, EfficientNet, smaller ViTs

• 8–12 GB: often enough
• 16 GB: plenty for most use cases
• 24 GB+: if you want larger batches or higher-res inputs

2) Vision Transformers (ViT), CLIP-style fine-tuning

• 12–16 GB: workable for smaller variants and modest batch sizes
• 24 GB: much easier
• 48 GB+: for larger ViTs or bigger images

3) Diffusion models / Stable Diffusion fine-tuning

This is usually the most memory-hungry common image fine-tuning case.

• LoRA / DreamBooth-style tuning:
  • 8–12 GB: possible with optimizations
  • 16 GB: practical and common
  • 24 GB: comfortable
• Full fine-tuning:
  • 24–48 GB+ often needed depending on resolution and batch size

What increases VRAM usage

• Higher image resolution
• Larger batch size
• Training the full model vs. LoRA/adapters
• Mixed precision vs. full precision
• Optimizer choice (Adam is memory-heavy)
• Gradient accumulation/checkpointing settings

Practical recommendation

If you’re setting up one GPU for general image fine-tuning:

• Minimum usable: 12 GB
• Recommended sweet spot: 16–24 GB
• Best if budget allows: 24 GB+

If you tell me:

1. the model type (CNN, ViT, Stable Diffusion, etc.),
2. the image resolution,
3. whether you’re doing full fine-tuning or LoRA,

…I can give you a much more exact VRAM estimate.