Skip to main content

Immutable artifacts

An immutable artifact is built once, identified by a tag that can never point at anything else, and promoted between environments by reference rather than rebuilt. In every edition of the BuiltForProd Baseline and its blueprints, every container image is pushed exactly once as main-<short sha>, releases add a vX.Y.Z tag to the same image, the registry refuses tag overwrites, and no pipeline pushes or references latest.

Why it matters in production​

"It worked in staging" is only meaningful if production runs the identical bytes. A rebuild for production can pick up a different base image, a different dependency resolution or a different commit. A moving tag such as latest can change under a running workload, so a restart deploys something nobody reviewed. Immutability makes a tag a fact: v1.4.2 is one build, forever, and the record of what ran in prod on a given day is trustworthy. The same reasoning applies to the tools that build the platform: a moved Git tag on a GitHub Action could run different code than the one reviewed.

How each edition implements it​

Registries that refuse to overwrite​

Each edition keeps its registry in the core-artifacts account, subscription or project, and each one makes a pushed tag permanent:

  • AWS Enterprise Baseline. The ecr unit creates every repository with repository_image_tag_mutability = "IMMUTABLE". Adding a tag to an existing image manifest still works; overwriting an existing tag fails. A lifecycle policy keeps every release image, ages out only the last 30 integration builds, and deletes untagged images after 14 days.
  • Azure Enterprise Baseline. Azure Container Registry in acme-core-artifacts locks each tag right after the push: every pipeline runs az acr repository update --write-enabled false --delete-enabled false on main-<sha> and on every release tag, so the tag can be neither overwritten nor deleted. On the Standard SKU a nightly registry task purges untagged manifests after 14 days.
  • GCP Enterprise Baseline. Artifact Registry in acme-core-artifacts has one repository per image with immutable_tags = true. Cleanup policies keep every v* release and the 30 most recent versions, delete main-* builds after 90 days and untagged images after 14 days; keep rules win over delete rules.

Build once, promote by tag​

The three workflows have the same names and roles in the web app code repository of every edition:

  • cd-integration.yml builds on merge to main and pushes a single tag, main-<short sha>. There is no latest alias; it would fail on the second push and is not used anywhere.
  • cd-release.yml runs when a GitHub Release is published. It reads the manifest of main-<short sha> for the released commit and puts the same manifest under the release tag, registry-side: docker buildx imagetools create on Azure, gcloud artifacts docker tags add on GCP. No rebuild happens. A re-run is a no-op because the tag already exists. If that commit was never built on main, the step fails: a release cannot introduce an image that was not tested in dev.
  • promote-prod.yml accepts only a release tag, checks it exists in the registry (proof it was built once and ran in staging), and opens the production pull request. Prod runs the same digest as staging.

Container images run as a fixed non-root user, and CI scans each image with Trivy, failing on fixable CRITICAL and HIGH vulnerabilities before the image is pushed.

When Terraform must know the tag​

The ETL trigger runs a container image, but its infrastructure unit names no tag. The module owns a value in the landing zone's store, creates it with bootstrap, ignores it afterwards, and reads it back at plan time to build the image reference. The pipelines write every tag they deploy there (main-<sha> for dev, the release tag for staging and prod):

EditionTriggerWhere the deployed tag is recorded
AWS Enterprise BaselineLambda functionSSM parameter /acme/usw2/<stage>/etl-trigger/image_tag
Azure Enterprise BaselineContainer Apps jobContract vault secret etl-trigger--image-tag
GCP Enterprise BaselineCloud Run serviceParameter Manager parameter etl-trigger--image-tag

The GCP edition records the web app front end's tag the same way, in frontend--image-tag. With immutable tags a moving latest cannot exist, and a fixed tag in Terraform would make every plan want to roll the function back to whatever the code says; recording the deployed tag where Terraform reads it keeps the plan equal to what is running.

The same rule for the toolchain​

InputAWS Enterprise BaselineAzure Enterprise BaselineGCP Enterprise Baseline
GitHub ActionsCommit SHA with the version in a commentCommit SHA with the version in a commentCommit SHA with the version in a comment
Registry modulesExact version, such as vpc 6.7.3None usedNone used
OpenTofu, Terragruntroot.hcl, TF_VERSION and TG_VERSIONroot.hcl, TF_VERSION and TG_VERSIONroot.hcl, TF_VERSION and TG_VERSION
Providershashicorp/aws ~> 6.65 in root.hclazurerm ~> 5.7, azuread ~> 3.10, azapi ~> 2.13 in root.hclgoogle and google-beta ~> 8.5 in root.hcl
SOPSSOPS_VERSION: "3.13.3"SOPS_VERSION: "3.13.3"SOPS_VERSION: "3.13.3"
tflintv0.64.0, AWS ruleset 0.48.0v0.64.0, tflint-ruleset-azurerm 0.32.0v0.64.0, tflint-ruleset-google 0.40.0

An action reference looks like actions/checkout@3d3c42e... # v7.0.1: the trailing comment carries the version for readers, the SHA carries the guarantee.

Upgrades are their own change with a plan review, as described in the versioning policy.

Worked example: rolling production back​

On the GCP edition, production is on v1.4.2 and the release is bad. Nobody rebuilds anything:

  1. API. Someone runs promote-prod.yml with v1.4.1. The workflow confirms the tag still exists in Artifact Registry (the cleanup policies keep every v* tag) and opens the prod pull request, which a person merges and then syncs in ArgoCD, as described under GitOps.
  2. Front end. Re-running the v1.4.1 release workflow deploys that release's front-end image as a new Cloud Run revision, in staging and, after the prod approval, in prod, and records the tag in frontend--image-tag.
  3. ETL trigger. Re-running the v1.4.1 release workflow of the ETL code repository deploys that tag and rewrites etl-trigger--image-tag, so the next infrastructure plan shows no change.

Every step deploys bytes that already ran in production once. On the AWS and Azure editions the same rollback is promote-prod.yml with the previous tag, and for the ETL trigger a re-run of the previous release, which rewrites the SSM parameter or the contract vault secret.

Common mistakes​

  • Pushing latest. The repository rejects the second push, and nothing in the platform reads it.
  • Re-tagging by rebuilding. A rebuild is a different artifact. Add the tag to the existing manifest, which is what cd-release.yml does.
  • Editing the ETL image-tag parameter by hand. The next infrastructure apply deploys that tag outside the pipeline; fix a wrong tag by re-running the release that should be live.
  • Changing the trigger's image in the console. On AWS the next apply rolls the Lambda function back to the recorded tag. On Azure and GCP the infrastructure ignores the running image of the Container Apps job or the Cloud Run service, so the console change stays until the next pipeline deployment, and meanwhile the recorded tag no longer describes what runs.
  • Pinning an action to a tag instead of a SHA. Tags move; the trailing comment carries the version for readers, the SHA carries the guarantee.