Architecture overview
The Secrets Blueprint keeps the secrets your people manage, such as third-party API keys and OAuth client secrets, in one Git repository, encrypted with SOPS under a separate Cloud KMS key for each stage. When a change merges, a GitHub Actions workflow becomes the service account sa-acme-secrets-syncer through Workload Identity Federation, decrypts the changed stage and writes each value as a Secret Manager secret named <app>--<KEY> in that stage's project, acme-plat-<stage>. The External Secrets Operator of the stage's GKE cluster turns the secrets into a Kubernetes Secret for the pod.
How it works
- An engineer edits a stage's file with SOPS. SOPS encrypts every value with that stage's Cloud KMS key before the file is saved, so Git only ever holds ciphertext.
- A pull request runs
Secrets Plan: one job per stage validates the files and dry-runs the sync, then comments what would change (Would create,Would update,Unchanged,Kept). The plan never contains a value. - Review follows the stage. Developers review
dev/files; staging and prod files need a platform lead or an infrastructure admin, and the staging and prod jobs wait for approval in their GitHub Environment. - On merge,
Sync Secrets to Secret Managerruns for the stages whose folder changed, one at a time. Each job signs in through Workload Identity Federation with no stored key, validates again and writes a new version of every secret whose value changed. - Workloads read the secrets at run time. In the Web App Blueprint, the External Secrets Operator reads them through the
ClusterSecretStorenamedgcp-smand refreshes them every hour.
The parts
| Part | What it is | Where it comes from |
|---|---|---|
| Secrets repository | One folder per stage, one encrypted YAML file per application, two scripts and two workflows | The Secrets Blueprint |
| SOPS keys | Key ring acme-sops in acme-core-security: sops-sandbox, sops-dev, sops-staging, sops-prod | The GCP Baseline |
| Syncer account | sa-acme-secrets-syncer in acme-core-security, the only identity the workflows use | The GCP Baseline |
| Secret Manager | One secret per key in each stage project acme-plat-<stage>, under one naming convention | Written by the sync |
| Consumers | The External Secrets Operator of each cluster, and any workload granted secret access | The Web App Blueprint |
The Baseline creates the keys and the syncer account; the blueprint brings the repository, the scripts and the workflows that use them. The GCP Baseline overview describes the projects these pieces live in.
Stages
The repository ships with the three stages the Web App Blueprint deploys: dev, staging and prod. That means three stage folders, three .sops.yaml rules, three plan jobs and three GitHub Environments. Sandbox is opt-in: it is added once the web app infrastructure deploys a sandbox stage, because until then no workload reads Secret Manager of acme-plat-sandbox. The Baseline already creates the sops-sandbox key and the syncer's access to the sandbox project, so opting in changes only the repository.
Who can touch which secrets
Access is enforced by Cloud KMS IAM, not only by GitHub. The grants on each stage's key decide who can decrypt that stage's files:
| Who | sandbox and dev | staging and prod |
|---|---|---|
| Platform leads and DevOps leads | Read and write | Read and write |
| Platform, DevOps, app, ETL and AI engineer groups | Read and write | None |
sa-acme-secrets-syncer (the workflows) | Decrypt only | Decrypt only |
A developer with write access to the repository still cannot read or change a production secret, because Cloud KMS refuses the decrypt. This is least privilege applied at the key.
Design choices
- Git is the record. Every change to a secret is a reviewed commit with an author, a date and an approver. Nobody edits a value in the console.
- One key per stage. Splitting by stage turns "who may see production secrets" into an IAM binding on a key, checked on every decrypt.
- Secret Manager is what workloads read. Applications never read the repository; each stage project holds only its own stage's values.
- No long-lived credentials. Both workflows exchange a GitHub token for a short-lived access token of the syncer; no key for the account exists, and a deny policy stops anyone from creating one.
- Changes only. A secret whose latest version already holds the value is left alone, so a sync adds a version only when a value changed.
Where to go next
What you receive lists the repository contents and the Baseline pieces the blueprint relies on. System architecture goes one level deeper.