Prompt
I’m building a platform engineering stack for multiple teams — how should Kubernetes fit in?
Latest observation
Kubernetes should usually be treated as the runtime substrate, not the platform itself.
In a platform engineering stack for multiple teams, Kubernetes fits best as the standardized execution layer that your platform exposes through opinionated abstractions, automation, and guardrails.
Where Kubernetes fits
Think of the stack in layers:
-
Infrastructure layer
- Cloud accounts / VPCs / networks / DNS / IAM
- Compute, storage, load balancers
- Often managed by Terraform/Pulumi/CloudFormation
-
Cluster layer
- Kubernetes clusters, node pools, autoscaling
- Add-ons like ingress controllers, cert-manager, CNI, CSI, policy engines
- Managed Kubernetes if possible
-
Platform layer
- Golden paths for deploying services
- CI/CD or GitOps
- Secrets, config, observability, service-to-service auth
- Templates, service catalogs, internal developer portal
- Guardrails: policy, quotas, namespaces, RBAC
-
Application layer
- Team-owned services and workloads
- Teams should not need to manage cluster internals
Kubernetes belongs mostly in layers 2 and 3, but your platform should make it feel like a simple, safe destination rather than a system each team has to understand deeply.
The right mental model
Kubernetes is good for:
- Multi-team standardization
- Isolated runtime environments
- Scheduling, self-healing, scaling
- Service discovery and traffic routing
- Policy enforcement
- Workload portability
- Building repeatable deployment paths
Kubernetes is not, by itself:
- A developer platform
- A workflow system
- An observability system
- A security model
- A deployment strategy
- A product experience for app teams
If teams interact directly with raw Kubernetes YAML all the time, you likely have a platform usability problem, not a Kubernetes problem.
How to use Kubernetes in a multi-team platform
1. Provide standardized deployment abstractions
Instead of asking teams to manage Deployments, Services, Ingress, HPA, ConfigMaps, etc. manually, give them:
- A service template
- A deployment module/chart
- A “create service” workflow
- A GitOps repo structure
- A small set of supported resource types
Example:
- Teams define:
- image
- CPU/memory requests
- ports
- env vars
- replicas
- Platform handles:
- namespaces
- ingress
- TLS
- autoscaling
- monitoring
- network policy
- rollout strategy
2. Use namespaces as the primary tenant boundary
For most organizations, namespaces are the right first-level isolation model:
- per team
- per environment
- or per service group
Apply:
- RBAC
- ResourceQuotas
- LimitRanges
- NetworkPolicies
- PodSecurity controls
- default service accounts
- admission policies
If a team needs stronger isolation, consider:
- separate clusters
- separate node pools
- separate accounts/projects
- stronger workload isolation with hardened policies
3. Keep cluster administration separate from app delivery
Platform engineers should own:
- cluster provisioning
- upgrades
- add-ons
- policy
- capacity management
- observability stack
App teams should own:
- their services
- their deployment config within approved boundaries
- their SLOs and app-level alerts
This separation avoids turning every team into partial Kubernetes admins.
4. Prefer GitOps for change management
For multi-team environments, GitOps works very well with Kubernetes:
- declarative desired state
- auditable changes
- easy environment promotion
- consistent rollback behavior
Common pattern:
- app teams merge config changes to Git
- GitOps controller syncs to clusters
- platform enforces policies and promotions
5. Make Kubernetes invisible where possible
The best platform experience often means teams don’t need to think about:
- node groups
- ingress controller details
- certificate management
- service mesh internals
- config maps vs secrets vs volumes
- rollout mechanics
They should think in terms of:
- “deploy this service”
- “expose this endpoint”
- “scale this workload”
- “add a queue consumer”
- “set CPU/memory”
- “view logs/metrics/traces”
What to standardize on top of Kubernetes
A strong platform on Kubernetes usually includes:
- Deployment path
- Helm/Kustomize/GitOps/operator-based workflows
- Identity
- workload identity, IRSA, SPIFFE, OIDC integration
- Secrets
- external secret manager integration
- Networking
- ingress, gateways, DNS, cert automation
- Observability
- logs, metrics, traces, dashboards, alerting
- Policy
- admission control, image signing, allowed registries, security contexts
- Cost and capacity
- quotas, rightsizing, autoscaling
- Developer portal
- service catalog, templates, ownership, docs, golden paths
When Kubernetes is a good choice
Kubernetes makes sense if you have:
- multiple teams deploying independently
- need for consistent runtime controls
- microservices or many services
- mixed workloads
- a desire to standardize deployment and operations
- platform engineers who can operate it well
It is often less valuable if you have:
- only a few services
- a small team with limited operational bandwidth
- mostly serverless or managed PaaS-friendly workloads
- no appetite for cluster/platform ownership
Common anti-patterns
1. “Every team manages its own cluster”
Usually too much operational burden and fragmentation.
2. “Raw Kubernetes is the developer interface”
Too complex; creates YAML sprawl and support load.
3. “Platform team owns everything in K8s”
Then app teams become dependent and slow.
4. “Kubernetes solves platform engineering”
It doesn’t. It’s one building block.
5. “We need Kubernetes because everyone else uses it”
Use it only if it fits your operating model and workloads.
A practical target architecture
For many organizations, a good end state looks like:
- One or a few shared managed clusters per environment or region
- Namespaces per team/service
- GitOps for deployments
- A golden-path service template
- Strong guardrails via policy
- Central platform-owned add-ons
- App teams own service config and code
- Internal developer portal to abstract complexity
Short recommendation
If you’re building a platform engineering stack for multiple teams, use Kubernetes as:
- the common runtime
- the standardization point
- the policy enforcement layer
But do not make it the primary interface for developers.
The platform should expose a simple product experience on top of Kubernetes:
- templates
- self-service workflows
- guardrails
- automation
- observability
- safe defaults
If you want, I can also sketch:
- a reference architecture for a multi-team Kubernetes platform, or
- a decision matrix for when to use Kubernetes vs serverless vs PaaS.