I need cheaper LLM inference than OpenAI

If you want cheaper LLM inference than OpenAI, the best route depends on what you mean by “cheaper”:

1) Cheapest option overall: self-host open models

If you have predictable volume, running an open model yourself is usually the lowest per-token cost.

Good options:

• Llama 3.1 / 3.2
• Qwen2.5
• Mistral Small / Nemo
• Gemma 2

Common serving stacks:

• vLLM
• TGI (Text Generation Inference)
• llama.cpp for CPU / quantized small models
• TensorRT-LLM for NVIDIA-heavy deployments

Pros:

• Lowest marginal cost at scale
• Full control over latency, privacy, and prompts
• No vendor lock-in

Cons:

• Ops complexity
• Need to manage infra, autoscaling, monitoring
• Quality may be lower than top proprietary models unless you use bigger models

2) Cheaper hosted APIs than OpenAI

If you want to avoid infra work, use providers that often undercut OpenAI:

• Anthropic sometimes competitive depending on model
• Google Gemini API can be cost-effective for some workloads
• Groq: very fast, often attractive for certain open models
• Together.ai
• Fireworks.ai
• DeepInfra
• OpenRouter as a broker to compare model prices
• Cerebras for some high-throughput use cases

Cheapest hosted tokens are often from:

• small open models
• batch/offline inference
• prompt caching
• quantized endpoints

3) Reduce cost without changing providers

If you stay with OpenAI or any provider, you can still cut cost a lot:

• Use a smaller model for most requests
• Add a routing layer:
  • simple requests → cheap model
  • hard requests → expensive model
• Shorten prompts
• Use RAG instead of dumping lots of context
• Use structured outputs to avoid long back-and-forth
• Cache:
  • prompt cache
  • response cache
  • embedding cache
• Use streaming + early stop when appropriate
• Fine-tune a small model for repetitive tasks

4) Practical recommendations

If you want the lowest bill and can operate infra:

• Start with vLLM + Llama 3.1 8B or Qwen2.5 7B
• Quantize if quality is acceptable
• Use a single GPU instance and benchmark cost per 1M tokens

If you want easy setup and lower cost than OpenAI:

• Try Together.ai, Fireworks.ai, or DeepInfra
• Compare with OpenRouter if you want multiple model choices

If latency matters a lot:

• Try Groq or Cerebras for supported models

5) Rule of thumb

• Best quality: OpenAI / Anthropic / Gemini top models
• Best cost-performance: hosted open models on Together/Fireworks/DeepInfra
• Lowest possible cost: self-hosted quantized open models

If you want, I can give you a shortlist ranked by cost, or a recommended stack based on your workload:

• chat app
• RAG
• coding assistant
• batch summarization
• agentic workflows