Prompt
I'm building Kubernetes services and want a sane way to rotate credentials
Latest observation
A good, sane pattern in Kubernetes is:
1) Keep credentials out of images and configs
Store secrets in a dedicated secret manager, not in container images, Helm values, or plain YAML in git.
Common options:
- Cloud secret managers: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault
- Kubernetes Secrets: acceptable for some cases, but weaker unless you add encryption at rest, RBAC, and good ops practices
2) Use short-lived credentials where possible
Best practice is to avoid static long-lived passwords entirely.
Examples:
- Workload identity / IAM roles for service accounts
- OIDC / federated auth
- mTLS with cert rotation
- Database auth with ephemeral tokens
- Service-to-service tokens with TTLs
If a credential can expire automatically, rotation becomes much easier.
3) Inject secrets at runtime, not build time
Use one of:
- Secret CSI Driver to mount secrets from your cloud secret manager
- External Secrets Operator to sync external secrets into Kubernetes Secrets
- Vault Agent Injector if using HashiCorp Vault
This lets pods get updated credentials without rebuilding images.
4) Design apps to reload secrets without restart
A lot of “rotation” pain comes from apps only reading secrets at startup.
Good patterns:
- Re-read files periodically
- Watch mounted secret files for changes
- Support SIGHUP or admin reload endpoint
- Use sidecar or operator to trigger reload/restart on secret change
If the app can’t hot-reload, use rolling restarts on secret updates.
5) Rotate with overlap
For credentials used by two parties, use a “dual valid” window:
- Create new credential
- Deploy/update consumers
- Verify new credential works
- Revoke old credential
This avoids downtime.
6) Automate rotation
Use automation, not manual runbooks.
Typical setup:
- Secret manager rotates secret on schedule
- Operator syncs updated secret into Kubernetes
- Deployment rolls pods or app reloads config
- Monitoring verifies success
7) Prefer per-service credentials
Don’t share one credential across many services.
Instead:
- one DB user per app/service
- one API client per service
- least privilege per workload
This limits blast radius and makes rotation safer.
8) Audit and monitor
Track:
- who accessed secrets
- when rotation happened
- whether old credentials are still in use
- failures after rotation
A practical Kubernetes recommendation
If you want a modern default:
- Use external secret manager
- Use External Secrets Operator or Secrets Store CSI Driver
- Use workload identity where possible
- Make apps reload secrets from mounted files
- Use rolling restart fallback for apps that can’t reload
Example rotation workflow
- Secret manager generates new DB password
- Operator updates Kubernetes secret or mounted file
- App reloads config or deployment rolls
- New connections use the new password
- After a grace period, revoke the old password
If you want, I can give you:
- a recommended architecture for AWS/GCP/Azure,
- a Kubernetes YAML example, or
- a rotation strategy for database credentials specifically.