Skip to main content

Kubernetes

Kubernetes is the container orchestrator: it schedules pods onto nodes, restarts them, scales them and gives them networking and identity. EKS provides the cluster; this page is about what the Web App Blueprint puts inside it and the rules those workloads run under.

What it does​

A pod is one or more containers scheduled together. A Deployment keeps a number of pod replicas running and rolls new images out gradually. A Service gives pods a stable address, an Ingress exposes them through a load balancer, a NetworkPolicy restricts their traffic, and a ServiceAccount is their identity. Namespaces partition all of this.

How BuiltForProd uses it​

Namespaces. Each stage's cluster has blueprint-app (created by the app-namespace unit), argocd, external-secrets, amazon-cloudwatch for Fluent Bit, and kube-system for the load balancer controller, ExternalDNS and the two autoscalers. Namespaces are created by OpenTofu, not by ArgoCD; the Application has CreateNamespace=false.

Where pods land. The cluster has two tiers of node. The EKS managed node group is the stable baseline that runs Kubernetes and the system-critical workloads. Karpenter provisions and removes capacity for application workloads. Cluster Autoscaler scales the managed node group itself. The baseline is tainted CriticalAddonsOnly=true:NoSchedule and every system component carries the matching toleration; the application chart sets no toleration, no node selector and no affinity, and that absence is the mechanism. A pod with nothing to tolerate the taint cannot land on the baseline, so it goes to Karpenter capacity, which carries no taint of its own.

Identity. The app-namespace unit creates the ServiceAccount blueprint-app-sa and an EKS Pod Identity association to a role that may read Parameter Store under /acme/usw2/dev/* and write CloudWatch logs. Every add-on except Fluent Bit uses Pod Identity too; Fluent Bit keeps IAM Roles for Service Accounts (IRSA) because its credential resolver cannot read the Pod Identity token.

The application pods, as the chart in acme-aws-blueprint-webapp-gitops defines them:

  • Run as the fixed non-root user 10001 with a read-only root filesystem, all capabilities dropped, no privilege escalation and the RuntimeDefault seccomp profile. An emptyDir at /tmp is the only writable path, for Gunicorn's heartbeat files.
  • Roll out with maxUnavailable: 0, so a new version never reduces capacity, and spread across zones and nodes with topology spread constraints.
  • Expose GET /health as the liveness probe and GET /ready, which checks DocumentDB and Redis, as the readiness probe.
  • Have a HorizontalPodAutoscaler in staging and prod, configured with a target CPU utilization and minimum and maximum replicas, and are protected by a PodDisruptionBudget during node drains. The cluster's metrics-server add-on supplies the CPU readings it acts on.
  • Sit behind a default-deny NetworkPolicy that allows only port 8080 in from the stage VPC CIDR, DNS to CoreDNS, 27017 and 6379 to the data stores, port 80 to the Pod Identity agent and 443 out to AWS APIs. The EKS VPC CNI add-on enforces it; the defense in depth page explains why the chart and the cluster are configured together.

Access. The Kubernetes and Helm providers in every add-on unit authenticate with aws eks get-token. Only the deployer role and the PlatformAdminAccess permission set have access entries, so kubectl works for administrators through SSO and for the pipeline, and for nobody else.

Terms you will see​

TermMeaning
NamespaceA partition of cluster objects, such as blueprint-app.
Security contextThe pod and container settings for user, filesystem and capabilities.
NetworkPolicyThe allow-list of ingress and egress for a set of pods.
HPAHorizontalPodAutoscaler, which sets a target CPU utilization and replica bounds.
PDBPodDisruptionBudget, the minimum pods kept running during voluntary disruption.
TaintA node marking that repels pods which do not carry the matching toleration.

Where to read more​