Immutable artifacts

An immutable artifact is built once, identified by a tag that can never point at anything else, and promoted between environments by reference rather than rebuilt. In the BuiltForProd platform every container image is pushed to ECR exactly once as main-<short sha>, releases add a vX.Y.Z tag to the same image, the registries reject tag overwrites, and no pipeline pushes or references latest.

Why it matters in production

"It worked in staging" is only meaningful if production runs the identical bytes. A rebuild for production can pick up a different base image, a different dependency resolution or a different commit. A moving tag such as latest can change under a running workload, so a restart deploys something nobody reviewed. Immutability makes a tag a fact: v1.4.2 is one build, forever, and the record of what ran in prod on a given day is trustworthy. The same reasoning applies to the tools that build the platform: a moved Git tag on a GitHub Action could run different code than the one reviewed.

How the platform applies it

Registries that refuse to overwrite

The ecr unit in core-artifacts creates every repository with repository_image_tag_mutability = "IMMUTABLE". Adding a tag to an existing image manifest still works; overwriting an existing tag fails. A lifecycle policy keeps every release image, ages out only the last 30 integration builds, and deletes untagged images after 14 days.

Build once, promote by tag

• cd-integration.yml builds on merge to main and pushes a single tag, main-<short sha>. There is no latest alias; it would fail on the second push and is not used anywhere.
• cd-release.yml runs when a GitHub Release is published. It reads the manifest of main-<short sha> for the released commit and puts the same manifest under the release tag. No rebuild happens. A re-run is a no-op because the tag already exists. If that commit was never built on main, the step fails: a release cannot introduce an image that was not tested in dev.
• promote-prod.yml accepts only a release tag, checks it exists in ECR (proof it was built once and ran in staging), and opens the production pull request. Prod runs the same digest as staging.

Container images run as a fixed non-root user, and CI scans each image with Trivy, failing on fixable CRITICAL and HIGH vulnerabilities before the image is pushed.

When Terraform must know the tag

The ETL Lambda runs a container image but the lambda-trigger unit names no tag. modules/lambda owns the SSM parameter /acme/usw2/<stage>/etl-trigger/image_tag, creates it with the value bootstrap, ignores its value afterwards, and reads it back at plan time to build the image URI. The pipelines write every tag they deploy to that parameter (main-<sha> for dev, the release tag for staging and prod). With immutable tags a moving latest cannot exist, and a fixed tag in Terraform would make every plan want to roll the function back to whatever the code says; recording the deployed tag where Terraform reads it keeps the plan equal to what is running.

The same rule for the toolchain

Input	How it is pinned
GitHub Actions	Commit SHA, with the release version in a trailing comment (actions/checkout@3d3c42e... # v7.0.1)
Registry modules	Exact version, such as terraform-aws-modules/vpc/aws?version=6.7.3
OpenTofu, Terragrunt, AWS provider	root.hcl and TF_VERSION / TG_VERSION in every workflow
SOPS	SOPS_VERSION: "3.13.3" in the secrets workflows
tflint	v0.64.0 in plan.yml, AWS ruleset 0.48.0 in .tflint.hcl

Upgrades are their own change with a plan review, as described in the versioning policy.

Worked example: rolling production back

Production is on v1.4.2 and the release is bad. Nobody rebuilds anything. For the web application, someone runs promote-prod.yml with v1.4.1; the workflow confirms the tag still exists in ECR (the lifecycle policy keeps release images indefinitely) and opens the prod pull request, which a person merges and syncs, as described under GitOps. For the ETL pipeline, re-running the release workflow for v1.4.1 re-deploys that tag to staging and, after approval, prod, and rewrites the SSM parameter so the next infrastructure plan shows no change. Both rollbacks deploy bytes that already ran in production once.

Common mistakes

• Pushing latest. The repository rejects the second push, and nothing in the platform reads it.
• Re-tagging by rebuilding. A rebuild is a different artifact. Add the tag to the existing manifest, which is what cd-release.yml does.
• Editing the ETL image-tag parameter by hand. The next infrastructure apply deploys that tag outside the pipeline; fix a wrong tag by re-running the release that should be live.
• Updating a Lambda function's image in the console. The next apply rolls it back to the recorded tag.
• Pinning an action to a tag instead of a SHA. Tags move; the trailing comment carries the version for readers, the SHA carries the guarantee.