How do I ensure my air-gapped inference workflow with an on-prem model is compliant?

To ensure an air-gapped inference workflow with an on-prem model is compliant, treat compliance as a process and evidence problem, not just a network setup problem. The exact requirements depend on your industry and jurisdictions, but the common themes are: data handling, access control, model provenance, logging, change control, and auditability.

Practical compliance checklist

1) Define the compliance scope

Identify which obligations apply:

• Privacy/data protection: GDPR, CCPA/CPRA, HIPAA, GLBA, etc.
• Security controls: ISO 27001, SOC 2, NIST 800-53, CIS, internal policies
• Industry rules: PCI DSS, FedRAMP, CJIS, export controls, SOX, etc.

Document:

• What data the model processes
• Who can access it
• Where it is stored and processed
• Whether any outputs are retained
• Whether the model is used for decisions affecting individuals

2) Classify the data

For an air-gapped system, the biggest compliance risk is usually sensitive data inside prompts, retrieved context, outputs, and logs.

You should classify:

• Inputs/prompts
• Retrieved documents or embeddings
• Model outputs
• Logs and traces
• Fine-tuning data, if any
• Temporary files, caches, crash dumps, checkpoints

Apply rules for:

• PII/PHI/PCI
• Confidential business data
• Secrets/keys
• Regulated records

3) Minimize data exposure

Only process the minimum necessary data.

• Redact or tokenize sensitive fields before inference if possible
• Avoid sending secrets or credentials into prompts
• Limit retrieval to the minimum relevant documents
• Use short retention windows for logs
• Prefer on-device/on-prem preprocessing for masking and classification

4) Control access tightly

Even air-gapped systems can be noncompliant if access is too broad.

Implement:

• Least privilege RBAC/ABAC
• MFA for admin access
• Separate admin, operator, and auditor roles
• Approval workflows for model updates and configuration changes
• Break-glass procedures with logging
• Periodic access reviews

5) Maintain strong provenance and supply-chain controls

For compliance, you need to know exactly what is running.

Keep records of:

• Model name, version, hash, source, and license
• Dependency list and hashes
• Container images and signatures
• Hardware/firmware versions
• Configuration baselines

Use:

• Signed artifacts
• Offline verification of checksums/signatures
• SBOMs for model-serving software
• Change tickets for every update

6) Log safely and auditably

Logs are often the hidden compliance issue.

Good practice:

• Log access, not sensitive content
• Redact prompts/outputs when possible
• Separate operational logs from security audit logs
• Protect logs from tampering
• Define retention and deletion schedules
• Ensure audit trails include who accessed what and when

Be careful that logs do not become a secondary data store for regulated data.

7) Validate the air gap operationally

“Air-gapped” should be verified, not assumed.

Document and test:

• No outbound network routes
• Disabled Wi-Fi/Bluetooth/cellular where applicable
• Controlled removable media process
• Physical port restrictions
• Firewall rules, switch configs, and segmented enclaves
• Malware scanning for any transferred artifacts

Include periodic inspections and configuration drift checks.

8) Manage updates and removable media securely

Since the system is isolated, updates are usually the compliance weak point.

Use a controlled media transfer process:

• Scan media on a non-production staging system
• Verify signatures and hashes offline
• Track chain of custody
• Record who transferred what and when
• Use approved update bundles only
• Keep rollback plans

9) Address model-specific risks

For LLMs and other generative models:

• Prevent prompt injection from retrieved content
• Restrict tool/function calling
• Validate outputs before downstream use
• Add policy filters for disallowed content
• Review whether the model can memorize or reproduce sensitive data
• If fine-tuned, document training data rights and privacy controls

If outputs drive decisions:

• Define human review thresholds
• Test for bias, error rates, and explainability requirements
• Keep records of validation and performance monitoring

10) Put retention and deletion policies in writing

Compliance usually requires knowing how long data persists.

Define:

• Prompt retention
• Output retention
• Audit log retention
• Backup retention
• Deletion method and schedule
• Legal hold process

Ensure the policy matches actual system behavior.

11) Create formal governance artifacts

You’ll want evidence such as:

• Data Protection Impact Assessment (DPIA) or equivalent
• Risk assessment
• System security plan
• Model card / AI use case documentation
• Access control policy
• Logging and retention policy
• Incident response plan
• Business continuity / disaster recovery plan
• Vendor/license review for model and dependencies

12) Test and audit regularly

Compliance is ongoing.

Conduct:

• Internal audits
• Vulnerability assessments
• Red-team/prompt-injection testing
• Access review audits
• Configuration compliance scans
• Backup/restore tests
• Incident drills

Record findings and remediation.

---

Common pitfalls

• Treating “air-gapped” as automatically compliant
• Logging prompts/outputs with sensitive data
• Using unlicensed or unvetted model weights
• Allowing uncontrolled USB/media transfers
• No evidence of change control
• Missing retention/deletion discipline
• No documented user access review
• Not considering outputs as regulated data
• Fine-tuning on data without proper rights/consent

---

A simple compliance approach

If you want a workable baseline:

1. Classify data and define allowed use cases
2. Restrict access and eliminate unnecessary retention
3. Verify model and software provenance
4. Harden the air gap and update path
5. Redact sensitive logs
6. Document governance, risk, and incident response
7. Audit regularly and keep evidence

---

If you tell me your industry and region — for example, “healthcare in the US” or “EU financial services” — I can map this to the likely specific regulatory controls and give you a more concrete checklist.