I’m building an internal developer platform for containerized apps — what Kubernetes setup makes sense?

For an internal developer platform (IDP) for containerized apps, the “right” Kubernetes setup usually depends less on raw cluster features and more on how much autonomy you want to give teams vs how much control/platform engineering wants to keep.

A good default is:

Recommended baseline

Use Kubernetes as the runtime, with a shared managed control plane and strong platform guardrails.

Typical structure

• Managed Kubernetes: EKS / GKE / AKS
• Separate clusters by environment at minimum:
  • dev / staging / prod
• Optionally separate clusters per tenant/team if isolation requirements are high
• Namespace per app/team inside each cluster
• GitOps for deployment delivery
• Ingress + cert automation
• Centralized observability
• RBAC + policy controls
• Autoscaling and resource quotas
• Self-service developer workflows through templates/CLI/portal

---

When this makes sense

This setup is a strong fit if you want:

• developers to deploy quickly without needing cluster admin access
• standardization across apps
• a path to multi-team growth
• secure separation between environments
• relatively low operational overhead

---

Core design choices

1) Cluster strategy

Best default: environment-separated shared clusters

• One cluster for dev/test, one for staging, one for prod
• Within each cluster:
  • namespace per app or team
  • resource quotas and network policies

Pros

• Lower cost and operational burden
• Easier to standardize
• Good for most internal platforms

Cons

• Less blast-radius isolation than per-team clusters
• Noisy neighbors possible without good quotas/limits

Use separate clusters per team or product if:

• strict compliance boundaries are required
• teams need custom Kubernetes versions/add-ons
• you have very different workload profiles
• one team’s failure must not affect others

A common hybrid is:

• shared non-prod cluster(s)
• dedicated prod cluster(s) for critical workloads

---

2) Namespace model

Use one namespace per application or one namespace per team/app combination.

Enforce:

• CPU/memory requests and limits
• ResourceQuota
• LimitRange
• Pod Security standards
• NetworkPolicies

This helps keep the platform safe while still enabling self-service.

---

3) Deployment model

Use GitOps rather than direct kubectl/apply from laptops.

Good options:

• Argo CD
• Flux

Benefits:

• auditable
• repeatable
• easy rollback
• aligns well with platform workflows

Usually pair this with:

• Helm charts, Kustomize, or a platform-specific template system
• opinionated app scaffolding

---

4) Developer experience

For an IDP, the Kubernetes setup should be hidden behind a thin self-service layer.

Common patterns:

• internal developer portal
• app scaffolding templates
• golden-path templates for services, jobs, cronjobs, workers
• deploy action from PR merge
• preview environments per PR if useful

If you want an off-the-shelf starting point, many teams use:

• Backstage for the portal
• Argo CD for deployment
• Crossplane or Terraform for infrastructure provisioning

---

5) Security and guardrails

Put controls at the cluster/platform layer so teams don’t need to think about them every day.

Recommended:

• RBAC with least privilege
• Pod Security Admission
• NetworkPolicies
• Secrets management via external secret store
• Image scanning
• Admission policies with Kyverno or OPA Gatekeeper
• Signed images if supply-chain security matters
• IRSA / workload identity for cloud permissions

---

6) Ingress and service exposure

Use a standard ingress layer:

• NGINX Ingress, Traefik, or cloud-native ingress controller
• cert-manager for TLS
• external-dns if you want automated DNS records

For service-to-service traffic, consider:

• starting without a service mesh unless you truly need it
• adding Istio/Linkerd later only if you need traffic shaping, mTLS, or advanced observability

---

7) Observability

Make this part of the platform, not an app-by-app DIY task:

• logs: Loki / ELK / cloud logging
• metrics: Prometheus + Grafana
• traces: OpenTelemetry + Tempo/Jaeger
• alerting: Alertmanager or cloud-native alerting

Standardize labels and dashboards early.

---

8) Autoscaling and capacity

Use:

• HPA for pods
• Cluster Autoscaler or Karpenter on AWS
• requests/limits as mandatory
• node pools by workload type if needed

For internal platforms, this prevents overprovisioning while preserving app-level elasticity.

---

A practical reference architecture

If I were designing a sensible IDP Kubernetes foundation, I’d do this:

Platform layer

• Managed Kubernetes (EKS/GKE/AKS)
• Terraform for cluster/network/provisioning
• Argo CD for GitOps
• Backstage for developer portal
• cert-manager, external-dns, ingress controller
• Prometheus/Grafana + centralized logs
• Kyverno/Gatekeeper for policy

Tenant/workload layer

• namespace per app/team
• per-namespace quotas, limits, and policies
• standard app Helm chart or Kustomize base
• secrets from external secret manager
• HPA enabled by default
• preview namespaces for PRs if needed

Environment model

• dev/staging shared
• prod either shared with stricter controls or isolated per critical domain/team

---

What I would avoid

• giving developers cluster-admin access
• running everything in one giant cluster with no quotas
• relying on manual kubectl deploys
• adding a service mesh on day one “just because”
• over-customizing the cluster before you have clear platform requirements

---

If you want the simplest workable setup

Start with:

1. One managed Kubernetes cluster for dev/staging
2. One managed Kubernetes cluster for prod
3. Namespace per app
4. Argo CD
5. Helm-based templates
6. Ingress + cert-manager
7. Prometheus/Grafana + logging
8. RBAC, quotas, network policies, pod security
9. Backstage or a similar portal for self-service

That gives you a strong, scalable IDP foundation without unnecessary complexity.

---

If you want, I can also give you:

• a minimal “starter stack”
• a reference architecture diagram
• or a decision matrix for shared-cluster vs per-team-cluster vs per-env-cluster.