Prompt
What should I use for secret rotation across Kubernetes and CI/CD?
Latest observation
For secret rotation across Kubernetes and CI/CD, the best choice depends on whether you want a single source of truth and how much you want to automate. In most modern setups, I’d recommend:
Best default: a secrets manager + workload identity
Use a centralized secrets manager such as:
- HashiCorp Vault
- AWS Secrets Manager
- Azure Key Vault
- Google Secret Manager
- 1Password Secrets Automation (nice for CI/CD, lighter-weight)
Then connect both Kubernetes and CI/CD to it using workload identity / short-lived auth, not long-lived static credentials.
Why this is the best pattern
- Centralized rotation policy
- Short-lived access for apps and pipelines
- Easier auditability
- No need to store the same secret in multiple places
- Better fit for automated rotation
Recommended architecture
For Kubernetes
Use one of these patterns:
-
External Secrets Operator
- Syncs secrets from an external manager into Kubernetes Secrets
- Good if your workloads expect native K8s Secrets
- Works well with AWS/GCP/Azure/Vault
-
Secrets Store CSI Driver
- Mounts secrets directly into pods
- Can avoid storing secrets in etcd
- Good for higher security requirements
-
Vault Agent Injector if using Vault
- Injects secrets into pods dynamically
- Strong option if you want dynamic secret generation and lease renewal
For CI/CD
Use:
- OIDC federation from your CI platform to the cloud/secrets provider
- Avoid storing cloud keys in the CI system
- Let jobs fetch secrets at runtime with short-lived credentials
Examples:
- GitHub Actions → AWS/GCP/Azure via OIDC
- GitLab CI → cloud provider OIDC or Vault
- Jenkins → OIDC or Vault AppRole only if needed, but AppRole is less ideal than OIDC
Rotation strategy
Rotate these types differently:
- Human credentials: use MFA + periodic rotation or remove entirely
- App secrets: rotate automatically via the secrets manager
- Database passwords / API keys: prefer dynamic secrets if possible
- Signing keys / certificates: use cert automation like cert-manager or managed PKI
Best practice
- Rotate secrets in the source system first
- Allow overlap during a grace period
- Roll workloads/pipelines to pick up new versions
- Revoke old versions after confirmation
Strong recommendations by scenario
If you’re on a major cloud
Use:
- Cloud-native secrets manager
- Kubernetes External Secrets or CSI Driver
- CI/CD with OIDC federation
This is usually the simplest and most maintainable.
If you’re multi-cloud or need advanced control
Use:
- HashiCorp Vault
- Kubernetes auth or cloud auth
- CI/CD via OIDC to Vault
Vault is the most flexible, especially for dynamic secrets and uniform policies across environments.
If you want something simpler for CI/CD only
Use:
- 1Password Secrets Automation
- Good developer experience
- Less operational overhead than Vault
- Strong for pipelines, less ideal as a full platform-wide secret backend
What I would avoid
- Storing secrets in Git
- Long-lived static cloud access keys in CI
- Manually rotating secrets by hand
- Duplicating secrets independently in Kubernetes and CI/CD without a source of truth
- Using sealed secrets alone as a rotation strategy
- Sealed Secrets helps with GitOps encryption, but it doesn’t solve rotation by itself
Practical recommendation
If you want a modern, solid setup:
Use Vault or your cloud provider’s secrets manager as the source of truth, then:
- Kubernetes: External Secrets Operator or CSI Driver
- CI/CD: OIDC federation to fetch secrets at job runtime
- Prefer dynamic secrets and short-lived credentials wherever possible
If you want, I can also give you:
- a best-choice matrix by cloud/platform, or
- a reference architecture for GitHub Actions + Kubernetes.