Skip to main content

Drift

Drift is the difference between what the code says an environment should be and what the environment is. In every edition of the BuiltForProd Baseline a scheduled workflow plans every account, subscription or project, opens a GitHub issue when a plan is not empty, and several parts of the platform are designed so that the common causes of drift cannot happen.

Why it matters in production​

Drift is how "it works in staging" becomes a lie. A console change made during an incident, a failed apply that left half a change, a hand-edited parameter: each one makes the next plan surprising, and a surprising plan is where people start clicking "approve" without reading. Found within a day, drift is a small revert. Found at the next deployment, it is an outage waiting for a trigger.

How each edition implements it​

Scheduled detection​

The mechanism is the same in every edition. drift-detection.yml in the Baseline repository runs at 01:00 UTC, one job per account, subscription or project folder. Each job runs terragrunt run --all plan -detailed-exitcode over environments/<ou>/<name>, which covers every region folder and global/ of that folder. Exit code 2 means the plan is not empty; the workflow then opens an issue titled Drift detected: <folder> with the plan output attached and the labels drift and infrastructure. Any other non-zero code means the plan itself failed, so that folder was not drift-checked at all: the workflow opens an issue titled Drift check failed: <folder>, which says so, and fails that folder's job so it shows red in the run list. The other folders keep running. Every drift plan runs with -lock=false, so it never blocks an apply. The cadence and the folder lists are tagged @optional: in the workflow.

EditionTuesday to FridayMonday and on demandIdentity
AWS Enterprise Baselinecore-network, core-identity, plat-dev, plat-staging, plat-prodAll 15 accountsRead-only: acme-aws-platform-baseline-reader, then acme-terraform-reader in each account
Azure Enterprise BaselineOff by default; an @optional schedule covers core-network, core-identity, core-security, plat-dev, plat-staging, plat-prodAll 15 subscriptionsid-acme-azure-platform-baseline-deployer in the prod Environment, so each run waits for one approval by the prod reviewers; a Reader identity cannot refresh the Key Vault secret and Entra units
GCP Enterprise Baselinecore-network, core-identity, plat-dev, plat-staging, plat-prodAll 15 project foldersRead-only: sa-acme-baseline-drift, outside any GitHub Environment and with no deployer impersonation

On the Azure and GCP editions the blueprint infrastructure repositories (web app and ETL) have a drift workflow of their own: one job per stage at 05:00 UTC, each in that stage's GitHub Environment with that stage's CI identity, opening issues the same way. Dev and staging run every day; prod runs on Mondays and on demand on Azure and only on demand on GCP, behind the prod reviewers in both. The workflows run on GitHub Actions; see the Azure and GCP explainers.

A person triages the issue. A console change is reverted by applying the code; a change that turns out to be wanted is put into the code, or imported into state, through a normal pull request. Either way the environment and the repository end up equal, which is the point of infrastructure as code.

Drift that cannot happen by design​

Several decisions remove classes of drift rather than detect them:

Source of driftHow the platform removes it
A moving image tagTags are immutable in every registry; the ETL trigger's deployed tag is recorded by the pipeline (SSM on AWS, the contract vault on Azure, Parameter Manager on GCP) and read back at plan time, so a plan never wants to roll the trigger back
Hand-written per-account copiesUnits are generated from one definition in every edition; nothing under environments/ is edited by hand
A region folder that disagrees with the address planThe ipam unit fails the plan on an unknown region or a mismatched slug, in every edition
Manual application changes in KubernetesArgoCD auto-syncs dev and staging with prune and self-heal; prod shows as out of sync until a person syncs
Resources created outside the pipeline with wrong tagsAWS: the organization tag policy reports them. Azure: Modify policies copy the tags from the resource group. GCP: the label constraint reports them in dry run
Console changes that should never happenService control policies, Azure Policy Deny and DenyAction, GCP organization policies and the IAM deny policy refuse them, as described under policy as code
A Defender plan switched in the portal (Azure)Every plan is written explicitly, Standard when on and Free when off, so a portal change shows up in the next plan

Drift after an upgrade​

The update lifecycle treats drift detection as the first check after any tool, provider or module upgrade: the next scheduled run confirms that the upgrade changed nothing it was not meant to. Layout-only changes are verified the same way before they merge, by rendering every generated unit before and after and comparing inputs, generated files, dependency paths and state keys; identical renders mean identical plans.

Worked example: a Tuesday morning issue​

On the Azure edition, at 01:00 UTC on a Monday the workflow plans every subscription folder; after one approval by the prod reviewers, the environments/core/security job exits with code 2. The issue Drift detected: environments/core/security shows a plan that sets one Microsoft Defender for Cloud plan back to Standard: during a cost review the Friday before, someone switched the plan to Free in the portal.

The engineer on triage reads the plan and checks with the security team. If the change was not wanted, a normal apply through a no-op pull request turns the plan back on, and the next run is clean. If it was wanted, the engineer sets that plan to false in security.hcl through a pull request, which the infra admins and the security team own and review, so that the subscriptions and the code agree. Either way the decision is recorded, and later changes reach the stages through promotion.

Common mistakes​

  • Fixing drift in the console. A console fix is more drift. Revert by applying the code, or change the code.
  • Closing a drift issue without a decision. Every issue ends in either an apply or a code change; leaving it open means the next plan is still surprising.
  • Editing the ETL image-tag parameter or envs/<stage>/values.yaml by hand. Both are written only by pipelines; a hand edit is drift the next run corrects or deploys.
  • Assuming a clean weekday run covers every account, subscription or project. Tuesday to Friday plans five AWS accounts or five GCP project folders (six Azure subscriptions when the optional Azure weekday schedule is on); the Monday run plans every account, subscription or project folder. Run the workflow on demand after a change to a core folder outside the weekday list.
  • Applying from a laptop to make a plan clean. The pipeline is the only writer; a local apply hides the cause and skips the record.