Skip to main content

Reliable

A Reliable system continues operating through expected failures: a zone goes down, a node drains, a database instance dies, a deployment rolls out. Every edition of the Baseline and its blueprints answers each of those with redundancy that is on by default where it is cheap and explicit, with its price beside the switch, where it costs money.

What every edition shares​

The application chart in the GitOps repository is the same in every edition, so the pod-level answers are too:

FailureMechanism
A pod or node is lost under loaddev 1 replica; staging a horizontal pod autoscaler from 2 to 5 at 70% CPU and a PodDisruptionBudget of 1; prod 3 to 10 at 60% CPU and a budget of 2; pods spread across zones and nodes
A deployment rolls outRolling updates with maxUnavailable: 0 and maxSurge: 1; the /ready probe checks both data stores before a pod receives requests
A node drains during reconciliationArgoCD runs highly available in staging and prod: two replicas of each component, a PodDisruptionBudget per component and redis-ha
Two applies collideApplies queue per Git ref and are never cancelled, and the state backend locks each unit's state
A dependency has not been deployed yetEvery dependency between units is mocked for validate and plan, so a whole environment plans end to end before anything exists; apply always uses real outputs (check-mock-outputs.py)

How each edition delivers it​

The editions differ in where the zone redundancy lives and which data stores carry the replicas.

AWS Enterprise Baseline​

Every VPC spans three Availability Zones, EKS nodes run across all private subnets and Transit Gateway attachments sit in every AZ. Cheap redundancy is on, and each data store is sized per stage in the Web App Blueprint's stack files.

FailureMechanism
The hub's NAT failsOne NAT Gateway per AZ in the hub VPC (nat_gateway_per_az = true), so an AZ failure does not stop every spoke's egress
A DocumentDB instance or its AZ failsProd runs three instances (a writer and two replicas across AZs) with deletion protection; staging two; dev one
A Redis node failsProd and staging run two nodes, multi-AZ with automatic failover; dev one
An application node is reclaimedKarpenter nodes carry the spot interruption queue, so a reclaim notice drains the node first

Designed exceptions, each an @optional: value: the Client VPN endpoint has one subnet association, and dev runs one DocumentDB instance, one Redis node and one replica of each ArgoCD component. The two extra NAT Gateways cost about $32 per month each, each extra DocumentDB db.t3.medium instance about $60 per month, each extra cache.t3.micro node about $12 per month, and ArgoCD HA one t3.medium node, about $30 per month per stage.

Azure Enterprise Baseline​

One list, firewall_zones = ["1", "2", "3"] in network.hcl, places Azure Firewall, its public IPs and the point-to-site VPN gateway's public IP across the three zones of eastus2. Every storage account (state, audit, shared artifacts, metastore) uses zone-redundant storage, with geo-zone-redundant storage as a switch per account, and the Container Apps environment of the self-hosted runners is zone-redundant.

FailureMechanism
A zone fails under the clusterThe AKS system pool spans zones 1 to 3 (Standard_D2as_v5, 2 to 3 nodes in dev, 3 to 4 in staging and prod); Node Auto Provisioning adds application nodes
A Cosmos DB node or its zone failsmongo_high_availability = true in staging and prod: a zone-redundant standby (ZoneRedundantPreferred) on tier M30
A Redis node failsredis_high_availability = true in staging and prod: two nodes of Azure Managed Redis replicated across zones
The cluster's ingress controller failsThe ArgoCD HA switch also raises the managed NGINX controllers to at least three replicas

Designed exceptions in dev: the Free AKS tier, Cosmos DB M10 without high availability, Redis Balanced_B0 on one node, Front Door Standard, and one replica of each ArgoCD component. The container registry runs the Standard SKU; Premium (about $50 per month) turns on zone redundancy. The stage files state the prices: a system node about $63 per month, the Standard AKS tier about $73 per month, M30 about $270 per month, and high availability doubles the Cosmos DB compute price.

GCP Enterprise Baseline​

Every stage runs in the home region on regional services spread across zones. The GKE cluster is regional, the Shared VPCs are global with regional subnets, and every VPC has its own Cloud NAT in each live region.

FailureMechanism
A zone fails under the clustergke_zones = 3 in staging and prod (2 in dev); the system pool runs 1 to 2 e2-standard-2 nodes per zone; node auto-provisioning places application nodes in the same zones
A Valkey node or its zone failsvalkey_replicas = 1 in staging and prod: Memorystore for Valkey spans zones with automatic failover, reached through Private Service Connect endpoints
A Firestore zone failsFirestore with MongoDB compatibility is a regional database in every stage
ArgoCD loses a nodeargocd_ha = true in staging and prod: two replicas each, redis-ha with three Redis pods behind HAProxy

Designed exceptions in dev: two zones, Valkey SHARED_CORE_NANO without a replica in one zone, and one replica of each ArgoCD component. The stage files state the system node price, about $49 per month per e2-standard-2 node, and that a Valkey replica doubles the node count.

Reliability in the delivery path​

Reliability is also about how change reaches what runs. In every edition, infrastructure changes are planned on every pull request and applied only after merge, in dependency order; application images are built once and promoted by tag, so prod runs exactly what staging ran; prod applies wait for the prod GitHub Environment; and prod GitOps pull requests are merged by a person and synced manually. The environments and promotion page follows a change through the stages, and the cost-aware convention, cheap resilience on and expensive resilience behind a priced switch, is the same one the Scalable page applies to growth.

How you verify it​

AWS Enterprise Baseline​

  • VPC console in a workload account: three private and three public subnets, one per AZ, and a Transit Gateway attachment in each; three NAT Gateways in the network account.
  • DocumentDB and ElastiCache consoles in prod: three DocumentDB instances with deletion protection; a two-node Redis replication group with multi-AZ and automatic failover.

Azure Enterprise Baseline​

  • az aks nodepool list --resource-group acme-eus2-prd-aks-rg --cluster-name acme-eus2-prd-aks: the system pool with zones 1, 2 and 3.
  • The Cosmos DB for MongoDB vCore cluster and the Azure Managed Redis cache in acme-plat-prod: high availability on.
  • The firewall in acme-core-network: zones 1, 2 and 3.

GCP Enterprise Baseline​

  • gcloud container clusters describe acme-usw1-prd-gke --region us-west1 --project acme-plat-prod: three node locations.
  • gcloud memorystore instances describe acme-usw1-prd-valkey --location us-west1 --project acme-plat-prod: one replica, multi-zone.

In every edition, kubectl get pdb,hpa -A against the staging or prod cluster shows the ArgoCD and application budgets and autoscalers, and the staging and prod stack files of the Web App Blueprint hold the sizing values above, each tagged @optional:. The Recoverable page covers what happens when a failure does get through.