Skip to main content

GitOps

GitOps means the desired state of a system lives in Git, every change is a pull request, and an automated process makes the running system match the merged state. In every edition of the BuiltForProd Baseline and its blueprints this holds for infrastructure (a merge to main applies the plan the pull request showed) and for applications (a deploy is a pull request against the GitOps repository that ArgoCD reconciles into the cluster).

Why it matters in production​

When Git is the only writer, every change has an author, a reviewer, a diff and a revert. Rollback stops being an emergency procedure and becomes git revert. Nobody needs cluster credentials or console access to ship, so credentials can be scoped to the pipeline and the reconciler. And because the reconciler keeps the cluster equal to the repository, a change made by hand is corrected rather than accumulated.

How each edition implements it​

The model is the same in the AWS, Azure and GCP editions; the names of the repositories carry the edition (acme-<cloud>-blueprint-webapp-code, acme-<cloud>-blueprint-webapp-gitops), and the registry is the edition's own.

Infrastructure: pull request, plan, merge, apply​

Every infrastructure repository runs plan.yml on pull requests and apply.yml on merge to main. The plan job runs the guards, formatting, tflint, Checkov and Trivy, then terragrunt run --all plan and posts the plan as a pull request comment. Code owners review the plan. On merge, the apply job applies every unit in dependency order behind a GitHub Environment gate. Applies queue per branch and are never cancelled, so two merges in quick succession cannot race on the same state. Both workflows trigger on everything a plan depends on: environments/, units/, stacks/, modules/, root.hcl, common.hcl, the address plan, the scripts and the workflow files themselves.

Applications: build once, deploy by pull request​

The web application's CI never touches a cluster. It builds an image, pushes it to the edition's registry (Amazon ECR, Azure Container Registry or Artifact Registry) under an immutable tag, and opens a pull request against acme-<cloud>-blueprint-webapp-gitops that sets image.repository and image.tag in blueprint-app/envs/<stage>/values.yaml. ArgoCD, running in each stage's cluster, watches that file and syncs.

StageOpened byMergeArgoCD sync
devcd-integration.yml (merge to main)auto-mergeautomatic, with prune and self-heal
stagingcd-release.yml (GitHub Release)auto-mergeautomatic, with prune and self-heal
prodpromote-prod.yml (manual, by release tag)a person reviews and mergesmanual

Nothing pushes to the GitOps repository directly. In every edition the three workflows share one composite action, .github/actions/gitops-pr, which mints a short-lived installation token for the acme-runner GitHub App (no deploy key, no personal access token), edits the values file with yq, pushes its own deploy/<stage>-<ref> branch and opens or reuses the pull request. Workflow permissions on the code repository stay contents: read.

The GitOps repository holds the Helm chart and per-stage values. Helm layers values.yaml, then values-<stage>.yaml, then envs/<stage>/values.yaml (the image). ArgoCD Application resources are created by the web platform's argocd unit, so the reconciler itself is infrastructure as code.

Per edition​

Only the registry, the release tagging and the front end differ:

EditionRegistry and release tagFront end
AWS Enterprise BaselineAmazon ECR with IMMUTABLE tags; the release adds its tag to the same imagecd-frontend.yml uploads to S3 and invalidates CloudFront
Azure Enterprise BaselineAzure Container Registry; cd-release.yml copies the manifest to the release tag registry-side, then locks the tag (write and delete disabled)cd-frontend.yml uploads to the stage's storage static website and purges the Front Door endpoint
GCP Enterprise BaselineArtifact Registry with immutable tags; cd-release.yml adds the tag with gcloud artifacts docker tags add, with no pull and no pushcd-frontend.yml deploys the front-end image as a Cloud Run revision in dev; cd-release.yml deploys the release to staging and, after approval, to prod

ArgoCD runs with argocd_auto_sync = true in dev and staging and false in prod in every edition. The ArgoCD explainers for the AWS, Azure and GCP editions describe each installation.

The ETL variant​

The ETL pipeline has no cluster, so it records instead of reconciling: after every deployment the code pipeline writes the deployed tag where the infrastructure reads it at plan time. On the AWS edition that is the SSM parameter /acme/usw2/<stage>/etl-trigger/image_tag, read by the lambda-trigger unit. On the Azure edition it is the contract vault secret etl-trigger--image-tag, and on the GCP edition the Parameter Manager parameter etl-trigger--image-tag; in both, the trigger module creates the value once with bootstrap and ignores later changes, so the pipeline owns it. The value is written only by the pipeline; a hand edit would make the next infrastructure apply deploy that tag outside the pipeline. The landing-zone contract page describes the stores.

Worked example: a hot fix reaches production​

An engineer merges a fix to main in the web app code repository. CI builds the image once and pushes main-a1b2c3d; the dev pull request auto-merges and ArgoCD rolls dev. A lead publishes release v1.4.3; cd-release.yml adds that tag to the same image and the staging pull request auto-merges. After verification, someone runs promote-prod.yml with v1.4.3; the prod GitHub Environment asks its required reviewers, the workflow checks that the tag exists in the registry (proof it was built once and ran in staging), and opens the prod pull request. An app lead and a DevOps lead merge it, then sync the application in ArgoCD. If the fix is wrong, they revert the merged deploy pull request or promote the previous tag. The artifact never changed between stages; see immutable artifacts. The story is the same in every edition: the registry is ECR, Azure Container Registry or Artifact Registry, and the front end follows through S3 and CloudFront, the storage static website and Front Door, or a Cloud Run revision.

Common mistakes​

  • Editing envs/<stage>/values.yaml by hand. The file is written only by deploy pull requests; a hand edit is overwritten by the next one and skips the review point.
  • Pushing to the GitOps repository directly. The design gives the workflows pull-request permissions through a GitHub App; a direct push bypasses the record.
  • Expecting prod to sync itself. Prod is argocd_auto_sync = false; after the merge, a person syncs. Dev and staging self-heal; prod does not.
  • Deploying with kubectl apply. ArgoCD reconciles dev and staging back to the repository, and prod shows as out of sync.
  • Forgetting "Allow auto-merge" on a fork. Dev and staging pull requests merge themselves only when the GitOps repository allows it.