How do I set up a private model gateway for autoscaling containerized deployments in an air-gapped environment?

A good way to think about this is: you need three things working together in the air gap:

1. A private model gateway to expose models behind one stable internal API.
2. An autoscaling container platform to run model-serving replicas.
3. An internal-only model artifact pipeline so images, weights, and dependencies never need public internet access.

Below is a practical reference architecture and setup path.

---

1) Recommended architecture

Inside the air gap:

• Private container registry
  Stores:

  • model-serving images
  • base images
  • sidecars
  • init containers
  • any inference runtime dependencies

• Model artifact store
  Stores:

  • model weights
  • tokenizer files
  • config files
  • prompts/templates
  • quantization artifacts

• Model gateway

  • Single internal endpoint for users/apps
  • Routes requests to the right model backend
  • Handles auth, rate limiting, logging, versioning

• Kubernetes or equivalent orchestrator

  • Runs model-serving pods
  • Supports Horizontal Pod Autoscaling or KEDA/custom metrics
  • Isolated ingress only inside the network

• Internal monitoring stack

  • Prometheus / Grafana / Loki or equivalent
  • Tracks request rate, latency, GPU utilization, queue depth, errors

• Internal DNS + TLS CA

  • Private certificates
  • Internal service discovery
  • No public certificate dependencies

---

2) Use an internal model gateway pattern

You can implement the gateway in one of two ways:

Option A: Dedicated gateway service

A small API service that:

• authenticates requests
• chooses a backend based on model name/version
• forwards requests to the correct inference service
• exposes a consistent OpenAI-style or REST API

This is usually best if you need:

• multiple models
• policy enforcement
• tenant isolation
• auditing
• version routing

Option B: Ingress directly to serving runtime

If your platform is simpler, you can route directly to:

• vLLM
• TGI
• Triton Inference Server
• KServe predictor endpoints

This is easier, but you lose some centralized control.

---

3) Build everything for offline use

Since you are air-gapped, prepare artifacts in a connected environment, then transfer them through your approved offline import process.

Prepare container images

Mirror all required images:

• model runtime image
• gateway image
• ingress controller
• metrics stack
• autoscaling components
• GPU operator if needed

Export them to tarballs or an internal registry bundle.

Example workflow:

• pull images in connected env
• scan/sign them
• export to archive
• import into private registry inside the air gap

Prepare model files

Bundle:

• model weights
• tokenizer
• config.json
• generation config
• code dependencies if using custom model code

Important:

• pin exact model revisions
• avoid runtime downloads
• set TRANSFORMERS_OFFLINE=1
• set HF_HUB_OFFLINE=1 if using Hugging Face-compatible tooling

---

4) Choose a serving runtime

Common choices:

vLLM

Good for:

• high-throughput LLM inference
• continuous batching
• OpenAI-compatible APIs

Hugging Face TGI

Good for:

• production text generation
• simple deployment
• decent autoscaling compatibility

Triton Inference Server

Good for:

• multi-model serving
• broader ML use cases
• GPU-optimized inference

KServe / Seldon / BentoML

Good for:

• standardized deployment abstractions
• easier scaling patterns
• multi-framework support

For LLMs, vLLM or TGI are common.

---

5) Deploy the gateway and model servers

A clean pattern is:

• Gateway service

  • internal HTTP endpoint
  • routes to one or more model services

• Per-model inference deployment

  • each model version runs as its own deployment or replica set
  • mounts model artifacts from PVC/object storage
  • uses GPU node selectors and tolerations if needed

• Internal service mesh or ingress

  • only accessible inside the cluster/network
  • mTLS if required

Example flow

Client → Gateway → Model router → Model server pods → Response

---

6) Autoscaling in an air gap

Autoscaling typically relies on one of these signals:

• CPU
• memory
• GPU utilization
• request queue depth
• request latency
• custom Prometheus metrics
• concurrency/in-flight requests

Best practice for model serving

For LLMs, CPU-only scaling is often poor. Prefer:

• request queue depth
• concurrent requests
• GPU utilization
• tokens/sec
• latency SLO breaches

Tools

• HPA for standard metrics
• KEDA for event/custom metric scaling
• Cluster Autoscaler if nodes can scale
• Static GPU pools if node scaling is not possible in the air gap

If your air-gapped environment cannot dynamically add nodes, you can still autoscale pods across a preprovisioned GPU pool.

---

7) Storage strategy for model assets

You need reliable local storage.

Common options:

• ReadOnlyMany PVCs from shared storage
• local PVs on GPU nodes
• internal object storage like MinIO or Ceph RGW
• NFS for simpler setups, though less ideal for high throughput

For large models:

• store weights in shared object storage
• mount or download from internal endpoint during pod startup
• use init containers to place artifacts on local disk if needed

Avoid pulling model files from the internet at startup.

---

8) Security controls

In an air-gapped gateway, security matters a lot.

Recommended controls

• Internal-only network exposure
• mTLS between gateway and model services
• RBAC for gateway admins
• API keys or JWTs for users/apps
• Audit logs for every request
• Rate limits and quotas per tenant
• Image signing and verification
• Vulnerability scanning before import
• Secrets stored in internal vault

Prevent accidental egress

• Kubernetes NetworkPolicies
• firewall rules
• no outbound DNS resolution to public resolvers
• proxy blocks if any egress path exists

---

9) Observability

Set up:

• Prometheus
  • GPU memory, GPU utilization, pod restarts, queue size, latency
• Grafana
  • dashboards for model performance and autoscaling behavior
• Central logs
  • request IDs, model version, tenant, latency, errors
• Tracing
  • gateway → backend traces if possible

Useful SLOs:

• p95 latency
• error rate
• time to first token
• tokens/sec
• saturation per GPU

---

10) A concrete deployment pattern

Components

• model-gateway Deployment
• llm-serving Deployment per model
• redis or queue if you need async/batching
• prometheus + keda
• private-registry
• minio or other internal artifact store

Request handling

1. Client sends request to model-gateway
2. Gateway authenticates and validates
3. Gateway forwards to the appropriate model endpoint
4. Serving pod processes request
5. Metrics emitted to Prometheus
6. KEDA/HPA scales replicas based on load

---

11) Implementation checklist

Air-gap prep

• Mirror container images
• Bundle model weights and configs
• Freeze dependency versions
• Generate SBOMs
• Scan and sign artifacts
• Import into internal registry/storage

Cluster setup

• Install Kubernetes or equivalent
• Configure GPU nodes and drivers
• Set up private ingress and DNS
• Configure internal TLS/CA
• Install monitoring stack

Gateway setup

• Choose gateway framework
• Define model routing rules
• Configure auth and quotas
• Add logging and tracing

Serving setup

• Build offline-serving image
• Mount model artifacts locally
• Disable all remote downloads
• Set resource limits and node selectors
• Enable batching if supported

Autoscaling setup

• Choose scaling metric
• Install HPA/KEDA
• Define min/max replicas
• Test load-based scaling
• Validate scale-down behavior

Security

• Network policies
• Secret management
• mTLS
• Image verification
• Audit logs

---

12) Suggested default stack

If you want a practical starting point:

• Kubernetes
• vLLM for serving
• Envoy or NGINX as the internal gateway/ingress
• KEDA + Prometheus for autoscaling
• MinIO for internal model artifacts
• Harbor or another private registry
• Vault for secrets
• Grafana/Loki/Prometheus for observability

This is a solid, modular setup for air-gapped deployments.

---

13) Common pitfalls

• Forgetting to mirror transitive dependencies
• Allowing runtime model downloads
• Using CPU metrics to scale GPU-bound inference
• Not reserving enough GPU memory headroom
• Assuming autoscaling nodes is possible when only pods can scale
• Missing TLS certs or internal DNS dependencies
• Not testing cold-start time with large models
• Not pinning exact model versions

---

14) Minimal example workflow

1. Build model-serving image in connected environment.
2. Download model weights and dependencies.
3. Package them into internal artifact storage.
4. Import artifacts into air-gapped registry/storage.
5. Deploy gateway and model server in Kubernetes.
6. Expose gateway only on internal network.
7. Attach Prometheus metrics.
8. Configure HPA/KEDA.
9. Load test and tune min/max replicas.
10. Lock down egress and audit logging.

---

If you want, I can also give you one of these next:

1. a reference Kubernetes YAML layout
2. a step-by-step offline build/import process
3. a sample model gateway design using vLLM or TGI
4. a KEDA autoscaling configuration based on Prometheus metrics