How do I choose between different downloadable models for private deployment and air-gapped inference?

For private deployment and air-gapped inference, the “best” downloadable model is usually the one that matches your hardware, latency, quality, and licensing constraints—not just the biggest or newest model.

Quick selection framework

1) Start with your deployment constraints

Ask:

• What hardware do I have?
  • CPU only
  • Single GPU
  • Multi-GPU
  • Edge device / laptop
• What is the max memory available?
  • Model weights + KV cache + runtime overhead
• What latency/throughput do I need?
  • Interactive chat
  • Batch jobs
  • Document processing
• What data sensitivity applies?
  • Full air-gap, no outbound access
  • Internal-only network
  • Audit requirements
• What languages/domains matter?
  • General purpose
  • Code
  • Multilingual
  • Legal/medical/finance

This determines the feasible model size and serving stack.

---

2) Choose the right model class

Common options:

• General chat/instruction model
  • Best for assistants, Q&A, summarization, extraction
• Code model
  • Better for software generation, debugging, refactoring
• Embedding model
  • For retrieval, search, RAG, clustering
• Reranker
  • Improves retrieval quality in RAG pipelines
• Vision-language model
  • For OCR-like tasks, image Q&A, document understanding

For most private deployments, you often need at least two models:

1. an embedding model for retrieval
2. a generator model for answering

---

3) Match model size to available memory

A rough rule:

• 7B–9B: good starting point for single GPU or strong CPU quantization
• 13B–14B: better quality, needs more memory
• 30B–40B+: higher quality but much heavier
• 70B+: best quality in many cases, but expensive to run privately

Also consider quantization:

• FP16/BF16: highest quality, highest memory use
• 8-bit: good compromise
• 4-bit: often best for private/local deployment
• GGUF / GPTQ / AWQ / EXL2: common formats for efficient local inference

If you’re air-gapped and resource-limited, a well-quantized smaller model often beats an unquantized larger one you can’t run efficiently.

---

4) Check licensing and redistribution terms

For private deployment, verify:

• Is commercial use allowed?
• Is on-prem / internal deployment allowed?
• Are there usage restrictions by industry or geography?
• Can you redistribute the weights internally?
• Do you need to pass along notices or attribution?

This is especially important if the model will be used by a company or in regulated environments.

---

5) Prefer models with offline-friendly ecosystems

Look for:

• Easy export/download of weights
• Stable checkpoints and versioning
• Support in offline runtimes:
  • llama.cpp
  • vLLM
  • Text Generation Inference (TGI)
  • TensorRT-LLM
  • Ollama
  • Transformers
• Documented quantized variants
• Known prompts/chat templates
• Good tokenizer support

For air-gapped environments, the easiest models are usually those with:

• widely supported architectures
• downloadable safetensors or GGUF
• minimal runtime dependencies

---

6) Evaluate on your own tasks, not benchmark tables alone

Benchmarks are useful, but real-world fit matters more. Test:

• factual accuracy
• hallucination rate
• instruction following
• tool-use behavior
• language quality
• retrieval grounding
• refusal/safety behavior if relevant
• latency and tokens/sec

A small pilot with your own data is often the best selector.

---

Practical recommendation by use case

If you need a general-purpose internal assistant

Pick:

• a 7B–14B instruction model
• quantized to 4-bit or 8-bit
• with support for your serving stack

This is usually the best balance of quality and operability for private deployments.

If you need high-quality long-form reasoning or complex summarization

Pick:

• a larger 30B+ model if hardware allows
• otherwise the best mid-sized model you can run reliably

If you need document search / RAG

Pick:

• a strong embedding model
• plus a generator model that handles citations and grounded answers well
• optionally a reranker

If you need code assistance

Pick:

• a code-specialized model
• test on your languages/frameworks
• ensure it handles long contexts if you’re using repo-level code tasks

If you need strict air-gap operations

Prioritize:

• offline installability
• small dependency footprint
• deterministic versioning
• local-only artifact storage
• no runtime calls to remote services

---

A simple decision rule

If you want the shortest path:

• Limited hardware → choose a small, well-quantized model
• Balanced private assistant → choose a 7B–14B instruct model
• Best quality and hardware is ample → choose a larger model
• RAG/search → add embeddings + reranking
• Air-gapped → favor open formats, simple runtimes, and offline docs

---

Final checklist

Before deciding, confirm:

• model fits in memory with headroom
• quantization is acceptable for your quality target
• license permits your use
• runtime supports offline inference
• tokenizer/chat template are documented
• model performs well on your real tasks
• update/patch process works without internet

If you want, I can also give you a practical model selection matrix for:

1. CPU-only servers
2. single-GPU private deployment
3. air-gapped enterprise clusters
4. RAG/document workloads