Prompt
I need a secret management approach for GitHub Actions, Terraform, and Kubernetes that won't break deploys. What should I do?
Latest observation
Use a single source of truth for secrets plus short-lived identity-based access everywhere. The safest approach that won’t constantly break deploys is:
Recommended pattern
1) Don’t store long-lived cloud keys in GitHub or Terraform state
- GitHub Actions: use OIDC federation from GitHub to your cloud provider.
- Terraform: use the same OIDC-backed identity for provisioning.
- Kubernetes: don’t put app secrets directly in Terraform state or plain K8s manifests.
This removes the biggest “deploy breaks because a secret rotated” problem.
2) Put real secrets in a dedicated secret manager
Use one of:
- AWS Secrets Manager / SSM Parameter Store
- GCP Secret Manager
- Azure Key Vault
- Or a platform-agnostic manager like HashiCorp Vault
Then:
- Store app secrets there
- Grant workloads access via workload identity / IAM / managed identity
- Sync secrets into Kubernetes only at runtime if needed
3) In Kubernetes, use External Secrets instead of committing secrets
Use one of these controllers:
- External Secrets Operator
- Secrets Store CSI Driver
- Vault Secrets Operator if using Vault
Pattern:
- Secret lives in cloud secret manager
- Kubernetes controller syncs it into a K8s Secret or mounts it as a volume
- Pods reference the secret normally
This avoids putting secret values in Git or Terraform state.
4) In GitHub Actions, use OIDC + minimal GitHub secrets
GitHub should store only:
- Non-sensitive config if necessary
- Maybe a bootstrap token if absolutely required
- Never cloud access keys unless there’s no alternative
Use:
permissions: id-token: write- cloud trust policy that allows only your repo/environment/branch
- temporary credentials from the cloud provider
This is the most deploy-safe setup because token issuance is dynamic and doesn’t depend on rotated static keys.
5) Make secret changes backward compatible
To avoid breaking deploys:
- Support two active versions during rotation
- Roll out code that can read both old and new if needed
- Rotate secrets gradually
- Restart only affected workloads
- Use versioned secret names or labels, not overwriting blindly
Example:
db-password-v1anddb-password-v2- or secret manager versioning with staged promotion
6) Separate “infrastructure secrets” from “application secrets”
Infrastructure secrets
Examples:
- Terraform backend creds
- Provider auth
- CI deploy identity
Best handled by:
- OIDC / managed identity
- no static secrets if possible
Application secrets
Examples:
- DB passwords
- API keys
- JWT signing secrets
Best handled by:
- secret manager + K8s sync at runtime
7) If you must use Terraform with secrets, be careful
Terraform can accidentally store secrets in:
- state
- plan output
- logs
Best practices:
- Avoid managing secret values directly in Terraform resources when possible
- Use Terraform to create secret containers/permissions, not raw secret contents
- If you must set a secret, mark variables
sensitive = true - Use remote encrypted state with tight access controls
- Don’t output secret values
8) Suggested reference architecture
GitHub Actions
- OIDC to cloud
- Terraform runs with temporary credentials
- Deploys app manifests
Terraform
- Creates:
- cluster
- IAM roles/service accounts
- secret manager entries/placeholders
- External Secrets Operator permissions
Kubernetes
- External Secrets Operator fetches secrets
- Pods use K8s Secret or mounted secret volume
- App reads secrets from env/volume at startup
9) A practical “won’t break deploys” rule set
- No static cloud keys in GitHub
- No plaintext secrets in Git
- No secret values in Terraform state if avoidable
- Secrets stored in a real secret manager
- Kubernetes pulls secrets at runtime
- Rotate with overlap and versioning
- Use least-privilege access per environment
10) If you want the simplest good setup
If you’re on AWS, I’d recommend:
- GitHub Actions → AWS OIDC
- Terraform → AWS OIDC
- Secrets in AWS Secrets Manager
- Kubernetes → External Secrets Operator + IRSA
- No long-lived AWS keys anywhere
That’s the cleanest “secure and deploy-safe” pattern.
If you want, I can give you:
- a concrete AWS-based implementation,
- a GitHub Actions OIDC example, or
- a Terraform + Kubernetes External Secrets example.