Drift
Drift is the difference between what the code says an environment should be and what the environment is. In the BuiltForProd AWS Baseline a scheduled workflow plans every account, opens a GitHub issue when a plan is not empty, and several parts of the platform are designed so that the common causes of drift cannot happen.
Why it matters in production
Drift is how "it works in staging" becomes a lie. A console change made during an incident, a failed apply that left half a change, a hand-edited parameter: each one makes the next plan surprising, and a surprising plan is where people start clicking "approve" without reading. Found within a day, drift is a small revert. Found at the next deployment, it is an outage waiting for a trigger.
How the platform applies it
Scheduled detection
drift-detection.yml in the landing zone runs at 01:00 UTC. Tuesday to Friday it plans the critical accounts: core-network, core-identity, plat-dev, plat-staging and plat-prod. On Monday, and on demand, it plans all fourteen account directories. Each job runs terragrunt run --all plan -detailed-exitcode --working-dir environments/<ou>/<account>, which covers every region folder and global/ of that account. Exit code 2 means the plan is not empty; the workflow then opens an issue titled Drift detected: <account> with the plan output attached and the labels drift and infrastructure. Any other non-zero code means the plan itself failed, so that account was not drift-checked at all: the workflow opens an issue titled Drift check failed: <account>, which says so, and fails that account's job so it shows red in the run list. The plan output is always written to the job log either way, and the other accounts keep running. The cadence and the account lists are tagged @optional: in the workflow.
A person triages the issue. A console change is reverted by applying the code; a change that turns out to be wanted is put into the code, or imported into state, through a normal pull request. Either way the environment and the repository end up equal, which is the point of infrastructure as code.
Drift that cannot happen by design
Several decisions remove classes of drift rather than detect them:
| Source of drift | How the platform removes it |
|---|---|
| A moving image tag | ECR tags are immutable; the ETL Lambda's deployed tag is recorded in SSM by the pipeline and read back at plan time, so a plan never wants to roll the function back |
| Hand-written per-account copies | Units are generated from one definition; nothing under environments/ is edited by hand |
| A region folder that disagrees with the address plan | The ipam unit's drift validation fails the plan on an unknown region, a mismatched slug or conflicting slugs |
| Manual application changes in Kubernetes | ArgoCD auto-syncs dev and staging with prune and self-heal; prod shows as out of sync until a person syncs |
| Resources created outside the pipeline with wrong tags | The organization tag policy reports non-compliant resources organization-wide |
| Console changes that should never happen | Service control policies deny them outright, as described under policy as code |
Drift after an upgrade
The update lifecycle treats drift detection as the first check after any tool, provider or module upgrade: the next scheduled run confirms that the upgrade changed nothing it was not meant to. Layout-only changes are verified the same way before they merge, by rendering every generated unit before and after and comparing inputs, generated files, dependency paths and state keys; identical renders mean identical plans.
Worked example: a Tuesday morning issue
At 01:00 UTC the workflow plans plat-staging and exits with code 2. The issue shows that a security group on the staging load balancer gained an inbound rule for a single office IP, added the previous evening during a debugging session. The DevOps engineer on triage reads the plan, confirms nobody intended to keep it, and runs the normal apply through a no-op pull request; the rule disappears and the next run is clean. Had the rule been wanted, the engineer would have added it to the unit's inputs in a pull request instead, so that staging and the code agree and prod can get the same rule through promotion.
Common mistakes
- Fixing drift in the console. A console fix is more drift. Revert by applying the code, or change the code.
- Closing a drift issue without a decision. Every issue ends in either an apply or a code change; leaving it open means the next plan is still surprising.
- Editing the ETL image-tag parameter or
envs/<stage>/values.yamlby hand. Both are written only by pipelines; a hand edit is drift the next run corrects or deploys. - Assuming a clean weekday run covers every account. Tuesday to Friday plans five accounts; the Monday run plans all fourteen. Run the workflow on demand after a change to a core account.
- Applying from a laptop to make a plan clean. The pipeline is the only writer; a local apply hides the cause and skips the record.