Reliable

A Reliable system continues operating through expected failures: an Availability Zone goes down, a node drains, a database instance dies, a deployment rolls out. The AWS Baseline and the blueprints answer each of those with redundancy that is on by default where it is cheap and explicit where it costs money.

Mechanisms by failure

Failure	Mechanism	Where
An Availability Zone fails	Every VPC spans three AZs; EKS nodes run across all private subnets; load balancers across all public subnets; Transit Gateway attachments in every AZ	Baseline VPC units, Web App Blueprint
The hub's NAT fails	One NAT Gateway per AZ in the hub VPC (nat_gateway_per_az = true), so an AZ failure no longer stops every spoke's egress and cross-AZ NAT charges disappear	network.hcl in the network account
A DocumentDB instance or its AZ fails	Prod runs three instances (a writer and two replicas across AZs) with deletion protection; staging runs two; dev one	Web App Blueprint documentdb unit, per-stage values
A Redis node fails	Prod and staging run two nodes, multi-AZ with automatic failover; dev one	Web App Blueprint elasticache unit
A node drains during reconciliation	ArgoCD runs in high-availability mode in staging and prod: two replicas of each component behind PodDisruptionBudgets and the chart's redis-ha; the node group minimum is three to satisfy its anti-affinity	Web App Blueprint argocd unit
A pod or node is lost under load	Replicas, a horizontal pod autoscaler and a PodDisruptionBudget per stage: dev 1 replica, no HPA; staging 2 replicas, HPA 2 to 5 at 70% CPU, minimum 1 available; prod 3 replicas, HPA 3 to 10 at 60% CPU, minimum 2 available; pods spread across zones and nodes	GitOps repository, values-<stage>.yaml
An application node is reclaimed	Karpenter's nodes carry the spot interruption queue, so a reclaim notice cordons and drains the node before EC2 takes it, and Karpenter launches a replacement for the evicted pods	Web App Blueprint karpenter unit
A deployment rolls out	Rolling updates with maxUnavailable: 0 and maxSurge: 1; readiness gates traffic; /ready checks DocumentDB and Redis before a pod receives requests	Helm chart, application code
Two applies collide	Applies queue per Git ref and are never cancelled; state is locked natively in S3	apply.yml, root.hcl
A dependency has not been deployed yet	Every dependency between units is mocked for validate and plan, so a whole environment plans end to end before anything exists; apply always uses real outputs	Unit definitions, check-mock-outputs.py

Designed exceptions

Some single points are left single on purpose, for cost, and each is named in the code:

• The Client VPN endpoint has one subnet association.
• Dev runs a single DocumentDB instance and a single Redis node.
• ArgoCD in dev runs one replica of each component.

Each is a per-stage value in a stack file, tagged @optional:, so raising it is a one-line change reviewed as a plan.

Reliability in the delivery path

Reliability is not only about what runs; it is about how change reaches it. Infrastructure changes are planned on every pull request and applied only after merge, in dependency order. Application images are built once and promoted by tag, so prod runs exactly what staging ran. Prod applies wait for the prod GitHub Environment, and prod GitOps pull requests are merged by a person and synced manually. The environments and promotion page follows a change through the stages.

What Reliable costs

The cost-aware convention is that cheap resilience is on and expensive resilience is a switch with its price beside it. The two extra NAT Gateways cost about $32 per month each. Each extra DocumentDB db.t3.medium instance is about $60 per month and each extra cache.t3.micro node about $12 per month. ArgoCD HA adds one t3.medium node per stage, about $30 per month. Those figures are in the code comments next to the switches; the Scalable page covers the growth side of the same knobs.

How you verify it

• VPC console in a workload account: three private and three public subnets, one per AZ, and a Transit Gateway attachment in each.
• VPC console in the network account: three NAT Gateways, one per AZ.
• DocumentDB and ElastiCache consoles in prod: three DocumentDB instances with deletion protection on; a two-node Redis replication group with multi-AZ and automatic failover on.
• kubectl get pdb,hpa -A against the staging or prod cluster: the ArgoCD and application budgets and autoscalers.
• The staging and prod stack files in the Web App Blueprint repository: the sizing values above, each tagged @optional:.

The Recoverable page covers what happens when a failure does get through. The hub-and-spoke networking page explains the routing that keeps a spoke reachable when a hub AZ fails.