What's the best self-hosted LLM stack for deploying open-weight models in a VPC with tenant isolation?

If you want the best self-hosted stack for open-weight LLMs in a VPC with tenant isolation, the strongest default choice today is:

Recommended stack

Core serving

• vLLM for inference serving
  • High throughput, good batching, OpenAI-compatible API
  • Strong support for modern open models
  • Good fit for GPU fleets in a VPC

Model gateway / API layer

• LiteLLM or a thin custom gateway
  • Unified OpenAI-compatible front door
  • Routing, auth, rate limits, quotas, model selection
  • Tenant-aware request logging and metering

Orchestration / deployment

• Kubernetes on your VPC
  • One namespace per tenant or per environment
  • GPU node pools isolated by tenant class
  • NetworkPolicies for east-west isolation
• Helm + Argo CD for repeatable deployment

Tenant isolation

Use a layered approach:

• Separate namespaces per tenant
• Dedicated service accounts + RBAC
• NetworkPolicies to block cross-tenant traffic
• Separate model deployments or even separate node pools for stricter tenants
• Per-tenant API keys / JWT claims at the gateway
• Per-tenant quotas, rate limits, and max context limits
• For high-sensitivity tenants: separate VPCs or separate clusters, not just namespace isolation

Storage / model registry

• S3-compatible object storage for model artifacts
• OCI registry or Hugging Face-style internal model registry for versioning
• Optional: MLflow or Bento-like registry pattern for governance

Observability

• Prometheus + Grafana
• OpenTelemetry
• Loki or ELK for logs
• Track:
  • tokens/sec
  • queue time
  • latency by tenant/model
  • GPU utilization
  • request failures
  • prompt/response audit trails if allowed

Security / governance

• OIDC integration for user auth
• mTLS internally if you need stronger service-to-service trust
• Secrets Manager / External Secrets Operator
• Admission controls for deployment validation
• Image signing / supply chain controls if regulated

---

Why this stack is the best default

vLLM

Best balance of:

• performance
• maturity
• OpenAI API compatibility
• support for continuous batching
• easy scaling across GPU nodes

Kubernetes

Best for:

• tenant isolation primitives
• autoscaling
• GPU scheduling
• network segmentation
• GitOps operations

LiteLLM

Useful if you need:

• a single API surface
• auth, routing, retries, fallbacks
• per-tenant policy enforcement
• model abstraction across vLLM/TGI/other backends

---

Good alternatives depending on your priorities

If you want maximum simplicity

• TGI (Text Generation Inference) + Kubernetes + API gateway
  • Very solid, especially for Hugging Face ecosystem
  • Slightly less flexible than vLLM in many deployments

If you want strong enterprise packaging

• NVIDIA NIM / Triton-based stack
  • Great if you’re heavily on NVIDIA infrastructure
  • More vendor-aligned, often less open-ended

If you want agent/tooling ecosystem in addition to serving

• Add LangGraph / Temporal / Celery on top
  • But keep serving and orchestration separate from inference layer

---

Tenant isolation patterns by sensitivity

Best-effort multitenancy

• Shared cluster
• Separate namespaces
• Shared GPU node pools
• Gateway-enforced auth and quotas

Strong multitenancy

• Shared cluster, but:
  • dedicated node pools per tenant tier
  • dedicated model deployments
  • strict NetworkPolicies
  • separate KMS keys per tenant

Highest isolation

• Separate VPC per tenant
• Separate cluster per tenant
• Separate storage and KMS boundaries
• No shared control plane components

For regulated environments, cluster-per-tenant is often the practical ceiling of secure isolation.

---

A concrete reference architecture

• Ingress
  • API Gateway / Envoy / NGINX
• Auth
  • OIDC/JWT
• Policy
  • LiteLLM gateway with tenant quotas and routing
• Serving
  • vLLM replicas on GPU nodes
• Orchestration
  • Kubernetes + Karpenter/Cluster Autoscaler
• Observability
  • Prometheus, Grafana, OTel
• Storage
  • S3-compatible model store
• Security
  • NetworkPolicies, RBAC, Vault/Secrets Manager, mTLS where needed

---

My short answer

If I had to pick one stack:

Kubernetes + vLLM + LiteLLM + OIDC/JWT + NetworkPolicies + per-tenant namespaces/node pools + Prometheus/Grafana

That’s the best balance of performance, operability, and tenant isolation for deploying open-weight models inside a VPC.

If you want, I can also give you:

1. a reference architecture diagram,
2. a recommended stack by budget/scale, or
3. a Terraform + Kubernetes deployment blueprint.