How do I remediated drift between desired and actual cloud state?

To remediate drift between desired and actual cloud state, you generally want a repeatable process that:

1. Detects drift
2. Determines what changed
3. Chooses the right remediation path
4. Restores the desired state safely
5. Prevents recurrence

1) Detect drift

Use your infrastructure-as-code and config management tools to compare declared state vs live state.

Common approaches:

• Terraform: terraform plan shows differences; terraform refresh is now generally handled implicitly by plan/apply in newer versions.
• Cloud-native tools:
  • AWS Config
  • Azure Policy
  • Google Cloud Asset Inventory / Config Controller
• Kubernetes:
  • GitOps tools like Argo CD or Flux detect and reconcile drift
• Security/compliance tools:
  • CSPM platforms can surface configuration drift

2) Identify the source of drift

Determine whether the drift came from:

• Manual console changes
• Emergency fixes/hot patches
• Automation bugs
• External systems modifying resources
• Misconfigured CI/CD pipelines

This matters because the remediation may be different if the drift was intentional vs accidental.

3) Classify the drift

Not all drift should be blindly reverted. Categorize it as:

• Safe to auto-remediate: e.g., tag changes, security group rule mismatch, missing labels
• Potentially disruptive: e.g., instance type changes, scaling differences, network route changes
• Intentional/approved: e.g., temporary mitigation during an incident

4) Remediate

Typical remediation methods:

A. Reconcile through IaC

If your infrastructure is managed by Terraform/CloudFormation/Bicep/etc.:

• Update the code to match the intended state
• Or revert the live environment by reapplying the code
• Run the pipeline in a controlled environment first if possible

Example pattern:

• Detect drift
• Review the plan
• Approve changes
• Apply the configuration

B. Use automated reconciliation

For systems like Kubernetes GitOps:

• The controller continuously reverts live state back to Git-defined state
• Useful for low-risk, frequently changing configs

C. Replace/rebuild resources

For immutable infrastructure or when state is badly corrupted:

• Recreate the resource from code
• Migrate traffic/data if needed
• Decommission the drifted resource

D. Patch manually, then codify

If the live change was necessary:

• Make the equivalent change in source control
• Redeploy through the pipeline
• Avoid leaving “snowflake” changes in production

5) Validate after remediation

After fixing drift:

• Re-run drift detection
• Confirm service health
• Check logs, metrics, and alerts
• Validate security/compliance posture

6) Prevent future drift

Good controls include:

• Restrict manual console access
• Use GitOps/IaC as the source of truth
• Require pull requests and approvals for infra changes
• Enable policy-as-code
• Use change windows and deployment pipelines
• Audit and alert on manual changes
• Lock critical resources where supported
• Tag resources with ownership and managed-by metadata

A practical remediation workflow

1. Detect drift with plan / policy tool / GitOps diff
2. Review whether drift is intended
3. Update code or plan to reflect desired state
4. Apply changes via pipeline
5. Validate
6. Add guardrails to prevent repeat drift

Important caution

If the drift affects:

• databases,
• networking,
• identity/access,
• encryption,
• production workloads,

then remediation should be change-controlled and may require backups, rollback plans, and maintenance windows.

If you want, I can also give you:

• a Terraform-specific drift remediation runbook
• a Kubernetes/GitOps version
• or a cloud-agnostic incident runbook template.