Recoverable
A Recoverable system treats failure as inevitable and engineers the way back: from a bad write, a deleted resource, a broken deployment, a corrupted state file or an out-of-band change. Backups you have not restored are assumptions, so every recovery path below is a mechanism in the code, not a promise.
Recovery by scenario
| What went wrong | How it is recovered | Mechanism |
|---|---|---|
| A bad infrastructure change was applied | Revert the merged pull request; the apply workflow applies the revert in dependency order | Git history, apply.yml |
| A bad application version reached dev or staging | Merge the next fix (a new main-<short sha>) or re-run the release for the previous version | Build-once, immutable tags |
| A bad application version reached prod | Run promote-prod.yml with the previous release tag; the tag still exists because tags are never overwritten. Or revert the merged deploy pull request in the GitOps repository | promote-prod.yml, ArgoCD |
| A bad ETL Lambda image reached a stage | Re-run the previous release; the deployed tag is recorded in SSM and never edited by hand, so the infrastructure plan agrees with what runs | cd-release.yml, SSM-tracked tag |
| Terraform state was corrupted or overwritten | Restore the previous object version; the state bucket keeps noncurrent versions for 90 days and is protected from deletion | S3 versioning, prevent_destroy |
| Two runs wrote state at once | Cannot happen: S3 native locking and applies queued per ref | use_lockfile, concurrency |
| A DocumentDB record was deleted or corrupted | Point-in-time restore from 7 days of automated backups; a final snapshot is taken on deletion; prod has deletion protection | DocumentDB backups |
| A data lake, front end, state or audit object was deleted or overwritten | Restore the previous version: the data lake keeps noncurrent versions 30 days, the state and audit buckets 90 days; current audit objects are never deleted | S3 versioning and lifecycle |
| Someone changed a resource in the console | Drift detection opens an issue with the plan; the fix is an apply from code or a deliberate update to the code | drift-detection.yml |
| A whole environment must be rebuilt | Every dependency is mocked, so an empty environment plans end to end; apply then runs in dependency order | Unit definitions with mocks |
| A hub Availability Zone or a data store node fails | Failover rather than recovery: NAT per AZ, multi-AZ DocumentDB and Redis in staging and prod | See Reliable |
State as critical data
Terraform state for every account and every blueprint lives in one bucket, acme-use1-root-tfstate, in the management account. It is versioned with 90-day noncurrent retention, encrypted, TLS-only, locked natively and marked prevent_destroy. Access is through a dedicated cross-account role. Each unit has its own state key derived from its path, so a bad run touches one unit's state, not the whole platform. Because the key strips the generated .terragrunt-stack/ segments, a unit keeps the same key whether a stack file lists it directly or a template generates it.
Rollback is a revert, not a procedure
Every environment gets its changes the same way, so rollback is the same operation everywhere:
| Stage | Infrastructure rollback | Application rollback |
|---|---|---|
| dev | Revert, merge, auto-apply | Revert, merge: a new build rolls out through the auto-merged pull request |
| staging | Revert, merge, auto-apply | Re-run the release for the previous version |
| prod | Revert, merge, prod Environment approval, apply, then validate | promote-prod.yml with the previous tag, merged by a person, synced manually |
The values files in the GitOps repository and the ETL image-tag parameter are never edited by hand; a rollback that bypassed the pipeline would be drift. The environments and promotion and immutable artifacts pages follow the forward path the rollback retraces.
Recovering from change itself
Some upgrades change where a unit lives or how routing is organized, and those have written runbooks in the Update Lifecycle workbook delivered with the repositories: moving a unit's state to a new key when a unit moves to a global folder, re-associating Transit Gateway attachments when isolation domains are introduced, enabling or disabling Network Firewall egress inspection (a brief egress pause), rolling the ETL Lambda image forward or back, and switching a security service on. After any update, drift detection on the next run, CloudWatch metrics and alarms an hour after apply and the next pipeline run confirm the platform is healthy.
Retention that recovery depends on
| Data | Retention |
|---|---|
| State bucket noncurrent versions | 90 days |
| Audit bucket noncurrent versions | 90 days; current objects never deleted, tiered to Standard-IA after 90 days and Glacier after 365 |
| Data lake noncurrent versions | 30 days; current objects tiered to Standard-IA after 90 days and Glacier after 180 |
| DocumentDB automated backups | 7 days |
| ECR images | Every release image per repository, plus the last 30 integration builds; untagged images deleted after 14 days |
How you verify it
- S3 console, state bucket in the management account: versioning enabled, lifecycle rule expiring noncurrent versions after 90 days.
- DocumentDB console in prod: backup retention 7 days, deletion protection enabled.
- The
driftissues in your landing-zone repository: each one is a recovery exercise with its plan attached. - The Update Lifecycle workbook in your handover documentation: the runbooks named above, each with expected plan output.
The drift page explains how to triage a drift issue, and the S3 explainer covers versioning and lifecycle rules.