Prompt
I'm building an enterprise IaC process with approvals, drift detection, and audit logs
Latest observation
Here’s a solid enterprise IaC blueprint for approvals, drift detection, and auditability. I’ll keep it vendor-neutral, but it maps well to Terraform/OpenTofu, Pulumi, Crossplane, or cloud-native tooling.
1) Core operating model
Use a separation of concerns model:
- Git = source of truth for desired state
- CI/CD = policy + validation + execution
- Cloud/infra provider = actual state
- Audit system = immutable record of decisions and actions
Typical flow:
- Developer opens a PR with IaC changes
- Automated checks run:
- formatting/linting
- security scanning
- policy checks
- plan generation
- Human approval is required
- Merge triggers controlled apply
- Drift detection runs on a schedule
- Exceptions and changes are logged immutably
2) Recommended approval workflow
A. Pull request gate
Require PRs for all changes to production-bound infrastructure.
PR checks:
fmt/ style check- static validation
- dependency/module lock verification
- policy-as-code validation
- secret scan
- infrastructure plan
Approval rules:
- 1–2 approvers minimum
- separate approvers for sensitive environments
- code owner review for critical modules
- break-glass path only for emergencies
B. Environment-based approval tiers
Use stronger controls as environment risk increases:
- Dev: automated merge + auto-apply
- Stage: one human approval
- Prod: two-person approval + change window + ticket reference
C. Change classification
Classify changes before apply:
- low risk: tag changes, security group additions with bounded scope
- medium risk: new service deployment, scaling adjustments
- high risk: IAM, networking, encryption, data stores, deletions
High-risk changes can require:
- security review
- SRE/platform approval
- CAB ticket or equivalent change record
- manual apply instead of auto-apply
3) Drift detection design
What to detect
Drift means actual infrastructure differs from declared state.
Common causes:
- manual console changes
- out-of-band automation
- provider-side defaults changing
- failed partial applies
- resource mutation by other teams
Drift detection cadence
- Critical prod: every 1–4 hours
- Standard prod: daily
- Non-prod: daily or on-demand
- Before apply: always run a refresh/plan check
Drift detection outputs
For each workspace/environment, detect:
- added resources
- deleted resources
- modified attributes
- provider/state version mismatches
Response workflow
- Drift detected
- Open incident or ticket automatically
- Notify owners in Slack/Teams/email
- Determine whether to:
- reconcile config to reality
- revert manual drift
- accept drift temporarily with exception
- Record resolution and timestamp
Important control
Do not silently auto-reconcile drift in prod unless explicitly approved.
Drift remediation should itself be a governed change.
4) Audit logging requirements
You want a tamper-resistant trail of:
- who proposed change
- who approved it
- what plan was reviewed
- what was applied
- when it ran
- what changed
- whether drift was found
- whether exceptions were used
Audit events to capture
- PR opened/updated/merged
- plan generated
- policy checks result
- approvals granted/denied
- apply started/completed/failed
- resources changed
- drift scan started/completed
- drift findings
- break-glass usage
- manual override/exception approval
Audit log characteristics
- append-only
- immutable storage or WORM-capable bucket
- time-synced timestamps
- identity-linked: human and service principal identities
- correlated across Git, CI, and cloud provider
Retention
Typical enterprise retention:
- 1 year minimum for operational logs
- 3–7 years for regulated environments
- separate retention for security/audit evidence
5) Policy and guardrails
Use policy-as-code to prevent unsafe plans from being applied.
Examples of policies:
- no public S3/blob/storage buckets
- no wildcard IAM policies
- encryption required for data stores
- allowed regions only
- mandatory tags
- no deletion of protected resources
- instance types restricted to approved families
- prod resources must use approved modules
Tools depend on stack, but the pattern is the same:
- evaluate plan JSON or resource graph
- fail the pipeline on policy violations
- allow exceptions only through controlled process
6) State management and access control
State
- centralize state per environment
- encrypt at rest
- lock state during runs
- restrict direct access to state files
- back up state and version it
Access
- least privilege for CI service accounts
- separate human from machine permissions
- read-only access for reviewers
- apply permissions limited to controlled pipelines
- break-glass credentials stored and monitored separately
Segmentation
Use separate:
- accounts/subscriptions/projects for dev/stage/prod
- state backends
- CI service principals
- approval groups
This limits blast radius.
7) Evidence and compliance
For compliance-heavy enterprises, store evidence for every change:
- PR link
- plan artifact
- policy evaluation output
- approval record
- apply log
- resulting diff
- drift scan result
- related ticket/change request
A good pattern is an evidence bundle per deployment, archived to immutable storage.
8) Recommended pipeline architecture
A practical CI/CD flow:
- Validate
- format
- syntax
- dependency checks
- Plan
- generate execution plan
- store as artifact
- Policy
- enforce guardrails
- Approval
- human approval via PR or release gate
- Apply
- only from protected branch/tag
- Post-apply verification
- smoke checks
- state refresh
- Audit archive
- push logs/artifacts to immutable store
- Drift detection
- scheduled or event-driven
9) Handling exceptions and break-glass
You need a formal exception process for emergencies.
Break-glass controls
- time-bound access
- MFA required
- separate credentials
- mandatory incident/ticket number
- automatic alerting when used
- mandatory post-incident review
Exceptions
Allow only with:
- documented risk acceptance
- expiration date
- owner
- compensating controls
- review and renewal process
10) Metrics to track
Useful KPIs:
- mean time to approve
- mean time to apply
- drift detection rate
- drift remediation time
- failed apply rate
- number of manual changes outside pipeline
- policy violation rate
- number of emergency overrides
- change failure rate
These help you tune the process without over-bureaucratizing it.
11) Minimal control set to start with
If you want a lean but enterprise-ready baseline:
- PR-based change control
- mandatory plan review for prod
- policy-as-code
- protected branches
- separate prod account/subscription
- scheduled drift detection
- immutable audit logs
- break-glass with monitoring
- state locking and encryption
12) Example governance model
Dev
- auto-plan, auto-apply
- drift detection daily
- logs retained 90 days
Stage
- plan + one approval
- apply after merge
- drift detection daily
- logs retained 1 year
Prod
- plan artifact required
- two approvals required
- change ticket required
- apply only via pipeline
- drift detection every 4 hours
- immutable logs retained 7 years
If you want, I can turn this into one of these next:
- a reference architecture diagram
- a Terraform/OpenTofu pipeline design
- a policy checklist for security/compliance
- a sample approval workflow in GitHub Actions / GitLab / Azure DevOps