Skip to main content

Karpenter

Karpenter is a Kubernetes node provisioner from AWS. It watches for pods the scheduler cannot place, launches an EC2 instance that fits them, and removes the instance again when the pods go. In the Web App Blueprint it supplies every node the application runs on.

What it does​

Karpenter has no node groups and no Auto Scaling groups. An EC2NodeClass describes what a node looks like: the AMI family, the IAM role, the subnets and the security groups. A NodePool describes what Karpenter may buy: architectures, capacity types, instance families and a hard resource limit. For an unschedulable pod the controller prices the offerings that satisfy its requests and launches the cheapest one that fits. Consolidation does the reverse, removing empty nodes and repacking under-used ones onto fewer or smaller instances.

How BuiltForProd uses it​

The karpenter unit installs the Karpenter chart (oci://public.ecr.aws/karpenter/karpenter, version 1.14.1, which is also the controller version) into kube-system with two controller replicas. Credentials come from an EKS Pod Identity association; the upstream submodule that ships with the EKS registry module also creates the node IAM role, its access entry, and the SQS queue that receives Spot interruption notices so a node is drained before EC2 reclaims it.

One EC2NodeClass and one NodePool, both named default, define the capacity:

SettingValue
ImageAmazon Linux 2023, latest release of the family; amd64 and Linux only
Instance familiesm6i, m7i, m6a, m7a; bare-metal sizes excluded
Capacity typesSpot and on-demand
vCPU ceiling32 in dev, 64 in staging, 128 in prod
ConsolidationEmpty or under-used nodes go after one minute; at most 10% disrupted at once
Node lifetime720 hours, after which the node is drained and replaced on a fresh AMI
Instance metadataIMDSv2 required, hop limit 1, so pods cannot reach the node's credentials

With both capacity types allowed, Karpenter takes Spot whenever Spot capacity exists for a type that fits and falls back to on-demand. The vCPU ceiling is the bill's upper bound: at the limit Karpenter stops launching and the excess pods stay Pending, so a runaway workload cannot scale without end. Each row is an @optional: value; the ceiling is set per stage in the stage stack file.

Subnets and security groups are selected by id, not by discovery tag: the subnets are the landing zone's private subnets from Parameter Store at /acme/usw2/dev/vpc/private_subnet_ids, and the security group is the cluster's node security group. No discovery tag is written to anything the landing zone owns.

Why application pods land here​

The managed node group is tainted CriticalAddonsOnly=true:NoSchedule and carries only the system components, each of which sets the matching toleration. The application chart sets no toleration, no node selector and no affinity, so its pods cannot schedule onto the baseline and Karpenter is the only capacity open to them. The NodePool deliberately applies no taint of its own, which keeps that door open. Cluster Autoscaler sizes the baseline underneath, and the two never contend because each touches only the nodes it owns.

An apply creates no instances. The first node appears when the first application pod does.

Terms you will see​

TermMeaning
NodePoolThe object listing what Karpenter may launch and how much of it.
EC2NodeClassThe object describing the AMI, role, subnets and security groups of a node.
ConsolidationRemoving or repacking under-used nodes to cut cost.
SpotSpare EC2 capacity at a discount that AWS can reclaim.
Interruption queueThe SQS queue carrying reclaim notices, so the node is drained in time.

Where to read more​