Skip to main content

Recoverable

A Recoverable system treats failure as inevitable and engineers the way back: from a bad write, a deleted resource, a broken deployment, a corrupted state file or an out-of-band change. Backups you have not restored are assumptions, so every recovery path below is a mechanism in the code of each Baseline edition, not a promise.

Recovery that works the same everywhere​

Every edition gets its changes through the same pipeline, so most recovery paths are the same operation in each:

What went wrongHow it is recoveredMechanism
A bad infrastructure change was appliedRevert the merged pull request; the apply workflow applies the revert in dependency orderGit history, apply.yml
A bad application version reached dev or stagingMerge the next fix (a new main-<short sha>) or re-run the release for the previous versionBuild-once, immutable tags
A bad application version reached prodRun promote-prod.yml with the previous release tag; the tag still exists because tags are never overwritten. Or revert the merged deploy pull request in the GitOps repositorypromote-prod.yml, ArgoCD
Someone changed a resource in the consoleDrift detection opens an issue with the plan; the fix is an apply from code or a deliberate update to the codedrift-detection.yml
A whole environment must be rebuiltEvery dependency is mocked, so an empty environment plans end to end; apply then runs in dependency orderUnit definitions with mocks
A zone or a data store node failsFailover rather than recoverySee Reliable

The values files in the GitOps repository and the recorded ETL image tag are never edited by hand; a rollback that bypassed the pipeline would be drift. The environments and promotion and immutable artifacts pages follow the forward path a rollback retraces.

StageInfrastructure rollbackApplication rollback
devRevert, merge, auto-applyRevert, merge: a new build rolls out through the auto-merged pull request
stagingRevert, merge, auto-applyRe-run the release for the previous version
prodRevert, merge, prod Environment approval, apply, then validatepromote-prod.yml with the previous tag, merged by a person, synced manually

How each edition delivers it​

State, data stores, the single-page app and the audit trail are recovered with each cloud's own mechanisms:

What went wrongAWS Enterprise BaselineAzure Enterprise BaselineGCP Enterprise Baseline
State corrupted or overwrittenPrevious S3 object version, kept 90 daysPrevious blob version, versions kept; 90-day soft deletePrevious object generation, kept 90 days; 7-day soft delete
Two runs wrote state at onceCannot happen: S3 native lockingCannot happen: a blob lease per state blobCannot happen: native locking in the bucket
A database record was lostDocumentDB point-in-time restore, 7 daysCosmos DB zone-redundant standby in staging and prod; delete lock in prodFirestore point-in-time recovery, 7 days; daily backups kept 35 days in prod
The front end is brokenPrevious S3 object version, or the previous releaseRe-run the previous release, which uploads its build and purges Front DoorRe-run the previous release, which deploys its Cloud Run revision
A bad ETL trigger imageRe-run the previous release; tag recorded in SSMRe-run the previous release; tag recorded in the contract vaultDeploy the previous tag; tag recorded in Parameter Manager
Audit data deleted or overwrittenVersioned bucket, current objects never deletedWrite-once storage for 365 days on every blob versionArchive bucket under a 365-day retention policy

AWS Enterprise Baseline​

Terraform state for every account and every blueprint lives in one bucket, acme-use1-root-tfstate, in the management account. It is versioned with 90-day noncurrent retention, encrypted, TLS-only, locked natively and marked prevent_destroy, and each unit has its own key derived from its path. DocumentDB keeps 7 days of automated backups, takes a final snapshot on deletion and has deletion protection in prod.

DataRetention
State bucket noncurrent versions90 days
Audit bucket noncurrent versions90 days; current objects never deleted, tiered to Standard-IA after 90 days and Glacier after 365
Data lake noncurrent versions30 days; current objects tiered to Standard-IA after 90 days and Glacier after 180
DocumentDB automated backups7 days
ECR imagesEvery release image per repository, plus the last 30 integration builds; untagged images deleted after 14 days

Some upgrades change where a unit lives or how routing is organized, and the Update Lifecycle workbook delivered with the AWS repositories has a runbook for each: moving a unit's state to a new key, re-associating Transit Gateway attachments when isolation domains are introduced, switching Network Firewall egress inspection, rolling the ETL Lambda image forward or back, and switching a security service on.

Azure Enterprise Baseline​

The state of every repository lives in stacmeeus2roottfstate, container tfstate, in acme-core-root. Blob versioning and the change feed keep every previous write, soft delete keeps deleted blobs and containers for 90 days, replication is zone-redundant, and prevent_destroy plus the CanNotDelete lock do-not-delete-tfstate refuse deleting the account or container. Shared keys are off; each blueprint identity may read, write and delete only the blobs under its own prefix.

DataRetention or protection
State accountEvery version kept; 90-day soft delete for blobs and containers
Audit storageWrite-once for 365 days (lock_immutability_policy makes it permanent); cool after 90 days, archive after 365; current blobs never deleted
Key Vaults90-day soft delete and purge protection; CanNotDelete locks on the Baseline vaults
Cosmos DB, Managed RedisCanNotDelete locks in prod; the module adds no backup setting of its own; Redis persistence is a switch
Data lake30-day blob and container soft delete
SPA storageVersioning and 7-day soft delete
Container registryTags locked against overwrite and deletion; untagged manifests purged after 14 days

A generated provider setting also stops a destroyed Key Vault from being purged and a resource group that still holds resources from being deleted.

GCP Enterprise Baseline​

The state of every repository lives in acme-usw1-root-tfstate in the seed project acme-core-root. Object versioning keeps every previous generation, a lifecycle rule deletes noncurrent versions after 90 days, soft delete keeps deleted objects 7 days, and prevent_destroy with force_destroy = false refuses deleting the bucket. Each repository may write only its own prefix, so a bad run damages one unit's state at most.

DataRetention or protection
State bucketNoncurrent versions 90 days; 7-day soft delete
Audit archiveRetention policy 365 days (lock_archive_bucket_retention makes it permanent); Nearline after 90 days, Coldline after 365; current objects never deleted
Central log bucket365 days (lock_log_bucket_retention)
FirestorePoint-in-time recovery 7 days in staging and prod; daily backups 35 days in prod; delete protection in prod
Memorystore for ValkeyDelete protection in prod
Data lake bucketsVersioning, 7-day soft delete, noncurrent versions 30 days; delete protection in prod
Artifact Registryv* releases kept, plus the 30 most recent versions; main-* builds deleted after 90 days, untagged after 14
Cloud KMS keysNever deleted

How you verify it​

AWS Enterprise Baseline​

  • S3 console, state bucket in the management account: versioning enabled, lifecycle rule expiring noncurrent versions after 90 days.
  • DocumentDB console in prod: backup retention 7 days, deletion protection enabled.

Azure Enterprise Baseline​

  • stacmeeus2roottfstate in acme-core-root, Data protection: versioning and 90-day soft delete on; Locks: do-not-delete-tfstate.
  • The audit storage account in acme-core-audit, Data protection: the version-level immutability policy of 365 days.

GCP Enterprise Baseline​

  • gcloud storage buckets describe gs://acme-usw1-root-tfstate: versioning, soft delete and the lifecycle rule.
  • Firestore in acme-plat-prod: point-in-time recovery enabled and the daily backup schedule.

In every edition, the drift issues of the Baseline repository are recovery exercises with their plan attached; the drift page explains how to triage one.