What's the most cost-effective way to run custom serving using an open-weight LLM?

The most cost-effective approach usually depends on your traffic pattern, latency needs, and model size, but in practice the best value often looks like this:

Best overall pattern

Use a smaller open-weight model, quantize it, and serve it on a single GPU with an efficient inference engine.

Typical stack

• Model: Llama 3 8B, Mistral 7B, Qwen2.5 7B, etc.
• Quantization: 4-bit or 8-bit (often 4-bit for lowest cost)
• Serving engine: vLLM, TGI, SGLang, or llama.cpp
• Hardware: one consumer or low-cost cloud GPU where possible
• Deployment: autoscaled container or always-on single node depending on traffic

If you want the cheapest possible setup

1. Choose a model that fits one GPU

A 7B–8B model is usually the sweet spot for cost/performance:

• Much cheaper to host than 30B+ models
• Often good enough for custom chat, extraction, support, and RAG
• Can run on:
  • 16–24 GB VRAM with quantization
  • sometimes even less with careful settings

2. Quantize aggressively

• 4-bit quantization lowers VRAM and cost a lot
• Tradeoff: slight quality drop, usually acceptable for serving
• Good options:
  • AWQ
  • GPTQ
  • GGUF for llama.cpp
  • bitsandbytes for some workflows

3. Use an efficient inference runtime

For throughput and batching:

• vLLM: usually the best default for serving many concurrent requests
• TGI: solid, production-friendly
• SGLang: strong for structured generation / agentic workloads
• llama.cpp: best when CPU or very low-cost deployment matters

Hardware choices by cost

Lowest infra cost

• Single GPU instance with an open-weight model
• Examples:
  • NVIDIA L4, A10, A100 if available
  • Consumer GPUs like RTX 4090 or 3090 can be very cost-effective if you can self-host

If traffic is low or spiky

• Use on-demand serverless GPU or scale-to-zero if cold-start latency is acceptable
• Cheaper than keeping a GPU always on

If traffic is steady

• One always-on GPU is often cheaper than serverless once utilization is decent
• Batch requests to increase GPU utilization

Important cost lever: batching

A lot of serving cost comes down to tokens per second per dollar. To improve that:

• Enable continuous batching
• Keep context lengths reasonable
• Cache prompts if possible
• Use smaller system prompts
• Avoid unnecessarily large output tokens

Rule of thumb

If your workload is:

• Low/medium volume: one quantized 7B–8B model on a single GPU
• High volume: vLLM/TGI on a more powerful GPU with batching
• Very cost-sensitive and simple tasks: llama.cpp on CPU or small GPU
• Need best quality: choose a larger model, but cost rises fast

Practical recommendation

If you just want the most cost-effective “default”:

1. Pick Qwen2.5 7B or Llama 3 8B
2. Serve with vLLM
3. Use 4-bit quantization if quality is acceptable
4. Run on a single L4 or A10-class GPU
5. Add autoscaling only if demand is bursty

If you want, I can also give you:

• a cheapest cloud deployment option comparison
• a self-hosted vs cloud cost breakdown
• or a recommended setup for your expected QPS / context length / latency target