How do I set up a model routing layer for auditing prompt and code usage across private repositories?

To set up a model routing layer for auditing prompt and code usage across private repositories, design it as a controlled gateway between your applications/developers and the model providers. The routing layer should:

1. Intercept all model calls
2. Log prompts, code, metadata, and responses
3. Apply policy controls
4. Route requests to the right model/provider
5. Store audit evidence securely

Here’s a practical blueprint.

---

1) Define the goals

Before building, decide what you need to audit:

• Prompt usage
  • Who sent it
  • Which repo/service it came from
  • Full prompt content or redacted version
  • Model used
  • Timestamp, latency, token usage
• Code usage
  • Whether source code was included in prompts
  • Which files/snippets were sent
  • Whether code was generated by the model
  • Correlation to commits/PRs/issues
• Compliance/security
  • PII/secrets detection
  • Retention policy
  • Access control for auditors
  • Tamper-evident logs

---

2) Put a gateway in the middle

All LLM traffic should go through a single endpoint, e.g.:

• llm-gateway.company.internal

Apps, CLI tools, IDE plugins, and automation should call the gateway instead of calling OpenAI/Anthropic/etc. directly.

Gateway responsibilities

• Authenticate caller
• Identify repo/team/user
• Enforce prompt logging policy
• Route to provider/model
• Capture response and usage metadata
• Store audit event
• Optionally block or redact sensitive content

---

3) Standardize the request schema

Use a common internal format, regardless of provider.

Example fields:

• request_id
• trace_id
• user_id
• repo_id
• repo_name
• branch
• commit_sha
• tool (IDE, CI, Slack bot, API)
• model_requested
• prompt_messages
• attachments
• code_context
• policy_tags
• timestamp
• client_ip
• auth_method

This makes auditing and routing easier.

---

4) Capture code provenance

For private repositories, don’t just log “code was sent” — capture where it came from.

Recommended metadata

• Repository URL/name
• Commit SHA
• File path
• Line ranges included
• Diff hunk or snippet hash
• Whether snippet was copied manually or extracted by tool
• Ticket/PR reference

Best practice

Instead of storing raw file contents only, also store:

• snippet_hash
• file_hash
• commit_sha
• paths[]

This allows traceability and later verification.

---

5) Add prompt and code redaction controls

Because private repos can contain secrets or sensitive IP:

• Run secret scanning on inputs
  • API keys
  • tokens
  • private certs
  • .env values
• Optionally redact:
  • passwords
  • customer data
  • authentication headers
• Maintain a “safe to log” / “restricted” classification

A good pattern:

• Store full content only in a highly restricted audit vault
• Store redacted content in normal logs
• Store content hashes for integrity and correlation

---

6) Route requests by policy

Routing logic can be based on:

• User role
• Repo classification
• Data sensitivity
• Request type
• Latency/cost
• Model availability

Examples:

• High-sensitivity repos → only approved on-prem or zero-retention providers
• Code completion → smaller/cheaper model
• Security-sensitive tasks → specialized model or human review
• Non-sensitive docs → general-purpose model

Example policy rules

• “Prod-source repos cannot be sent to external providers unless redacted”
• “Secrets detected ⇒ block and alert”
• “Legal repo content ⇒ route to approved internal model only”
• “Use model X for code generation, model Y for summarization”

---

7) Make logs tamper-evident

For auditing, logs should be immutable or append-only.

Good options

• Write to append-only object storage
• Use WORM / retention locks
• Send to a SIEM with immutability controls
• Hash-chain events (event_hash includes previous event hash)
• Sign events with service key

Audit event fields

• event_id
• timestamp
• caller_identity
• repo_id
• request_payload_hash
• response_payload_hash
• model_provider
• model_name
• policy_decision
• redaction_applied
• token_count_in/out
• latency_ms

---

8) Store prompts and responses separately from logs

A common architecture:

• Operational logs: metadata only
• Audit vault: encrypted full prompts/responses/snippets
• Analytics store: aggregated metrics, redacted text, hashes

This reduces risk and limits exposure.

Suggested storage controls

• Envelope encryption
• Per-tenant/per-repo keys
• Strict RBAC
• Time-bound access
• Audit access to the audit logs themselves

---

9) Build developer and repo identity mapping

To know where code/prompt came from, integrate with:

• SSO/IdP (Okta, Entra ID, Google Workspace)
• Git provider (GitHub Enterprise, GitLab, Bitbucket)
• CI/CD identity
• IDE plugin identity
• Service accounts

Map every request to:

• human user
• bot/service account
• repo
• branch
• commit
• environment

---

10) Support common entry points

The routing layer should cover all ways people use models:

• Web UI
• API clients
• IDE extensions
• CLI tools
• CI/CD jobs
• ChatOps bots
• Internal automation

If even one bypass exists, the audit trail is incomplete.

---

11) Add observability and reporting

Useful dashboards:

• Requests by repo/team/user
• Top models used
• Prompts containing code
• Secret detections
• Blocked/redacted requests
• Cost by repo
• Tokens per project
• Audit exceptions and bypass attempts

Alerts:

• Secret detected in prompt
• New repo added without policy
• Direct provider call outside gateway
• Unusual usage spike
• Large code snippet exported

---

12) A reference architecture

Flow

1. Developer or service sends request to gateway
2. Gateway authenticates caller
3. Gateway enriches request with repo/user/commit metadata
4. DLP and policy engine inspect prompt/code
5. Gateway decides:
   • allow
   • redact
   • block
   • route to model A/B/C
6. Request sent to provider
7. Response returned
8. Audit event persisted
9. Metrics exported to observability stack

Components

• API Gateway / LLM proxy
• Policy engine: OPA, Cedar, custom rules
• DLP/secret scanner
• Model router
• Audit log store
• Encrypted vault
• Analytics/BI layer
• SIEM integration

---

13) Minimal implementation pattern

If you want a lightweight start:

Step 1

Create an internal API:

• /v1/chat
• /v1/completions
• /v1/embeddings

Step 2

Require headers:

• X-User-Id
• X-Repo-Id
• X-Commit-SHA
• X-Policy-Tier

Step 3

Log:

• request hash
• redacted request body
• response hash
• model/provider
• token counts

Step 4

Run secret scanning on inbound prompt content

Step 5

Persist full payloads in encrypted storage with access controls

---

14) Example audit record

{
  "event_id": "evt_12345",
  "timestamp": "2026-07-20T12:34:56Z",
  "user_id": "u_8842",
  "repo_id": "repo_payments_api",
  "commit_sha": "a1b2c3d4",
  "model_requested": "gpt-4.1",
  "model_routed": "gpt-4.1",
  "policy_decision": "allow",
  "redactions": ["api_key", "email"],
  "request_hash": "sha256:...",
  "response_hash": "sha256:...",
  "token_usage": {
    "input": 1832,
    "output": 412
  },
  "latency_ms": 842,
  "source": {
    "tool": "ide-plugin",
    "client_version": "1.8.0",
    "ip": "10.14.2.7"
  }
}

---

15) Security and privacy cautions

• Don’t log secrets in plaintext in general-purpose logs
• Restrict access to raw prompts/responses
• Set retention limits
• Classify repos by sensitivity
• Make users aware of logging and purposes
• Follow legal/compliance rules for employee monitoring and data retention
• Consider a “do not store” mode for especially sensitive workflows

---

16) Recommended policy capabilities

Your routing layer should support:

• Allow/deny rules
• Data classification
• Redaction transforms
• Model/provider allowlists
• Repo-based exceptions
• User-based exceptions
• Human approval for high-risk requests
• Audit export for compliance

---

17) If you want an open-source-ish stack

A common stack could be:

• Proxy: FastAPI / Go / Envoy
• Policy: Open Policy Agent
• Secrets scanning: trufflehog / gitleaks / custom regexes
• Storage: Postgres for metadata, S3 for encrypted blobs
• Search/analytics: OpenSearch/Elasticsearch, BigQuery, Snowflake
• SIEM: Splunk, Sentinel, etc.
• Tracing: OpenTelemetry

---

18) Practical rollout plan

1. Start with a single gateway for one team
2. Log metadata only
3. Add redaction and secret detection
4. Expand to all repo-connected tools
5. Add policy-based routing
6. Add immutable audit storage
7. Add compliance reporting and alerting

---

If you want, I can also give you:

• a sample architecture diagram
• a FastAPI/Go gateway skeleton
• an OPA policy example
• a database schema for audit logging