What should I use for LLM chat in production?

If you mean “what should I build with for a production chat experience,” the safest default is:

Use a hosted, production-grade model API with structured chat support, tool/function calling, streaming, and strong observability.
For most teams, that means OpenAI, Anthropic, or Google Gemini rather than self-hosting to start.

Practical recommendation

If you want the simplest path to production

• OpenAI GPT-4.1 / GPT-4o class models for general chat
• Add:
  • streaming
  • tool/function calling
  • conversation state in your backend
  • logging/monitoring
  • guardrails and moderation
  • retry/fallback logic

If you care most about very strong reasoning / safety style

• Anthropic Claude models are often a strong choice for assistant-like chat.

If you are already on Google Cloud or need multimodal + ecosystem integration

• Gemini can be a good fit.

What matters more than the model

In production, the model is only part of the solution. You usually want:

• Conversation memory stored in your database, not in the prompt forever
• RAG for company knowledge instead of stuffing everything into context
• Tool use for actions like search, database queries, ticket creation, etc.
• Prompt/version management so you can iterate safely
• Eval suite with real test conversations
• Rate limiting, auth, and abuse prevention
• Fallbacks if the primary model fails or degrades

When to self-host

Self-hosting open models like Llama 3.x or Mistral can make sense if:

• you need data residency / strict compliance
• you have very high volume and can optimize infra
• you need full control over latency/cost tradeoffs

But self-hosting usually means more work:

• model serving
• scaling
• GPU cost
• quality tuning
• safety filtering
• upgrades

Good “production chat” stack

A common stack looks like:

• Frontend: React/Next.js chat UI
• Backend: Node.js / Python API
• LLM API: OpenAI / Anthropic / Gemini
• Memory: Postgres + Redis
• Vector search: pgvector / Pinecone / Weaviate / Milvus
• Observability: traces + prompt logs + user feedback
• Safety: moderation, allowlists for tools, PII handling

My blunt advice

• Start with a managed API model.
• Do not self-host first unless compliance forces you to.
• Build around tool use, retrieval, and evals, not just raw prompting.
• Have at least one fallback model/provider.

If you want, I can give you a specific recommendation by use case:

1. customer support bot
2. internal knowledge assistant
3. coding assistant
4. high-scale consumer chat
5. regulated enterprise environment