Recoverable
A Recoverable system treats failure as inevitable and engineers the way back: from a bad write, a deleted resource, a broken deployment, a corrupted state file or an out-of-band change. Backups you have not restored are assumptions, so every recovery path below is a mechanism in the code of each Baseline edition, not a promise.
Recovery that works the same everywhere
Every edition gets its changes through the same pipeline, so most recovery paths are the same operation in each:
| What went wrong | How it is recovered | Mechanism |
|---|---|---|
| A bad infrastructure change was applied | Revert the merged pull request; the apply workflow applies the revert in dependency order | Git history, apply.yml |
| A bad application version reached dev or staging | Merge the next fix (a new main-<short sha>) or re-run the release for the previous version | Build-once, immutable tags |
| A bad application version reached prod | Run promote-prod.yml with the previous release tag; the tag still exists because tags are never overwritten. Or revert the merged deploy pull request in the GitOps repository | promote-prod.yml, ArgoCD |
| Someone changed a resource in the console | Drift detection opens an issue with the plan; the fix is an apply from code or a deliberate update to the code | drift-detection.yml |
| A whole environment must be rebuilt | Every dependency is mocked, so an empty environment plans end to end; apply then runs in dependency order | Unit definitions with mocks |
| A zone or a data store node fails | Failover rather than recovery | See Reliable |
The values files in the GitOps repository and the recorded ETL image tag are never edited by hand; a rollback that bypassed the pipeline would be drift. The environments and promotion and immutable artifacts pages follow the forward path a rollback retraces.
| Stage | Infrastructure rollback | Application rollback |
|---|---|---|
| dev | Revert, merge, auto-apply | Revert, merge: a new build rolls out through the auto-merged pull request |
| staging | Revert, merge, auto-apply | Re-run the release for the previous version |
| prod | Revert, merge, prod Environment approval, apply, then validate | promote-prod.yml with the previous tag, merged by a person, synced manually |
How each edition delivers it
State, data stores, the single-page app and the audit trail are recovered with each cloud's own mechanisms:
| What went wrong | AWS Enterprise Baseline | Azure Enterprise Baseline | GCP Enterprise Baseline |
|---|---|---|---|
| State corrupted or overwritten | Previous S3 object version, kept 90 days | Previous blob version, versions kept; 90-day soft delete | Previous object generation, kept 90 days; 7-day soft delete |
| Two runs wrote state at once | Cannot happen: S3 native locking | Cannot happen: a blob lease per state blob | Cannot happen: native locking in the bucket |
| A database record was lost | DocumentDB point-in-time restore, 7 days | Cosmos DB zone-redundant standby in staging and prod; delete lock in prod | Firestore point-in-time recovery, 7 days; daily backups kept 35 days in prod |
| The front end is broken | Previous S3 object version, or the previous release | Re-run the previous release, which uploads its build and purges Front Door | Re-run the previous release, which deploys its Cloud Run revision |
| A bad ETL trigger image | Re-run the previous release; tag recorded in SSM | Re-run the previous release; tag recorded in the contract vault | Deploy the previous tag; tag recorded in Parameter Manager |
| Audit data deleted or overwritten | Versioned bucket, current objects never deleted | Write-once storage for 365 days on every blob version | Archive bucket under a 365-day retention policy |
AWS Enterprise Baseline
Terraform state for every account and every blueprint lives in one bucket, acme-use1-root-tfstate, in the management account. It is versioned with 90-day noncurrent retention, encrypted, TLS-only, locked natively and marked prevent_destroy, and each unit has its own key derived from its path. DocumentDB keeps 7 days of automated backups, takes a final snapshot on deletion and has deletion protection in prod.
| Data | Retention |
|---|---|
| State bucket noncurrent versions | 90 days |
| Audit bucket noncurrent versions | 90 days; current objects never deleted, tiered to Standard-IA after 90 days and Glacier after 365 |
| Data lake noncurrent versions | 30 days; current objects tiered to Standard-IA after 90 days and Glacier after 180 |
| DocumentDB automated backups | 7 days |
| ECR images | Every release image per repository, plus the last 30 integration builds; untagged images deleted after 14 days |
Some upgrades change where a unit lives or how routing is organized, and the Update Lifecycle workbook delivered with the AWS repositories has a runbook for each: moving a unit's state to a new key, re-associating Transit Gateway attachments when isolation domains are introduced, switching Network Firewall egress inspection, rolling the ETL Lambda image forward or back, and switching a security service on.
Azure Enterprise Baseline
The state of every repository lives in stacmeeus2roottfstate, container tfstate, in acme-core-root. Blob versioning and the change feed keep every previous write, soft delete keeps deleted blobs and containers for 90 days, replication is zone-redundant, and prevent_destroy plus the CanNotDelete lock do-not-delete-tfstate refuse deleting the account or container. Shared keys are off; each blueprint identity may read, write and delete only the blobs under its own prefix.
| Data | Retention or protection |
|---|---|
| State account | Every version kept; 90-day soft delete for blobs and containers |
| Audit storage | Write-once for 365 days (lock_immutability_policy makes it permanent); cool after 90 days, archive after 365; current blobs never deleted |
| Key Vaults | 90-day soft delete and purge protection; CanNotDelete locks on the Baseline vaults |
| Cosmos DB, Managed Redis | CanNotDelete locks in prod; the module adds no backup setting of its own; Redis persistence is a switch |
| Data lake | 30-day blob and container soft delete |
| SPA storage | Versioning and 7-day soft delete |
| Container registry | Tags locked against overwrite and deletion; untagged manifests purged after 14 days |
A generated provider setting also stops a destroyed Key Vault from being purged and a resource group that still holds resources from being deleted.
GCP Enterprise Baseline
The state of every repository lives in acme-usw1-root-tfstate in the seed project acme-core-root. Object versioning keeps every previous generation, a lifecycle rule deletes noncurrent versions after 90 days, soft delete keeps deleted objects 7 days, and prevent_destroy with force_destroy = false refuses deleting the bucket. Each repository may write only its own prefix, so a bad run damages one unit's state at most.
| Data | Retention or protection |
|---|---|
| State bucket | Noncurrent versions 90 days; 7-day soft delete |
| Audit archive | Retention policy 365 days (lock_archive_bucket_retention makes it permanent); Nearline after 90 days, Coldline after 365; current objects never deleted |
| Central log bucket | 365 days (lock_log_bucket_retention) |
| Firestore | Point-in-time recovery 7 days in staging and prod; daily backups 35 days in prod; delete protection in prod |
| Memorystore for Valkey | Delete protection in prod |
| Data lake buckets | Versioning, 7-day soft delete, noncurrent versions 30 days; delete protection in prod |
| Artifact Registry | v* releases kept, plus the 30 most recent versions; main-* builds deleted after 90 days, untagged after 14 |
| Cloud KMS keys | Never deleted |
How you verify it
AWS Enterprise Baseline
- S3 console, state bucket in the management account: versioning enabled, lifecycle rule expiring noncurrent versions after 90 days.
- DocumentDB console in prod: backup retention 7 days, deletion protection enabled.
Azure Enterprise Baseline
stacmeeus2roottfstateinacme-core-root, Data protection: versioning and 90-day soft delete on; Locks:do-not-delete-tfstate.- The audit storage account in
acme-core-audit, Data protection: the version-level immutability policy of 365 days.
GCP Enterprise Baseline
gcloud storage buckets describe gs://acme-usw1-root-tfstate: versioning, soft delete and the lifecycle rule.- Firestore in
acme-plat-prod: point-in-time recovery enabled and the daily backup schedule.
In every edition, the drift issues of the Baseline repository are recovery exercises with their plan attached; the drift page explains how to triage one.