Skip to main content

Architecture overview

The Secrets Blueprint keeps the secrets your people manage, such as third-party API keys and OAuth client secrets, in one Git repository, encrypted with SOPS under a separate Cloud KMS key for each stage. When a change merges, a GitHub Actions workflow becomes the service account sa-acme-secrets-syncer through Workload Identity Federation, decrypts the changed stage and writes each value as a Secret Manager secret named <app>--<KEY> in that stage's project, acme-plat-<stage>. The External Secrets Operator of the stage's GKE cluster turns the secrets into a Kubernetes Secret for the pod.

How it works​

  1. An engineer edits a stage's file with SOPS. SOPS encrypts every value with that stage's Cloud KMS key before the file is saved, so Git only ever holds ciphertext.
  2. A pull request runs Secrets Plan: one job per stage validates the files and dry-runs the sync, then comments what would change (Would create, Would update, Unchanged, Kept). The plan never contains a value.
  3. Review follows the stage. Developers review dev/ files; staging and prod files need a platform lead or an infrastructure admin, and the staging and prod jobs wait for approval in their GitHub Environment.
  4. On merge, Sync Secrets to Secret Manager runs for the stages whose folder changed, one at a time. Each job signs in through Workload Identity Federation with no stored key, validates again and writes a new version of every secret whose value changed.
  5. Workloads read the secrets at run time. In the Web App Blueprint, the External Secrets Operator reads them through the ClusterSecretStore named gcp-sm and refreshes them every hour.

The parts​

PartWhat it isWhere it comes from
Secrets repositoryOne folder per stage, one encrypted YAML file per application, two scripts and two workflowsThe Secrets Blueprint
SOPS keysKey ring acme-sops in acme-core-security: sops-sandbox, sops-dev, sops-staging, sops-prodThe GCP Baseline
Syncer accountsa-acme-secrets-syncer in acme-core-security, the only identity the workflows useThe GCP Baseline
Secret ManagerOne secret per key in each stage project acme-plat-<stage>, under one naming conventionWritten by the sync
ConsumersThe External Secrets Operator of each cluster, and any workload granted secret accessThe Web App Blueprint

The Baseline creates the keys and the syncer account; the blueprint brings the repository, the scripts and the workflows that use them. The GCP Baseline overview describes the projects these pieces live in.

Stages​

The repository ships with the three stages the Web App Blueprint deploys: dev, staging and prod. That means three stage folders, three .sops.yaml rules, three plan jobs and three GitHub Environments. Sandbox is opt-in: it is added once the web app infrastructure deploys a sandbox stage, because until then no workload reads Secret Manager of acme-plat-sandbox. The Baseline already creates the sops-sandbox key and the syncer's access to the sandbox project, so opting in changes only the repository.

Who can touch which secrets​

Access is enforced by Cloud KMS IAM, not only by GitHub. The grants on each stage's key decide who can decrypt that stage's files:

Whosandbox and devstaging and prod
Platform leads and DevOps leadsRead and writeRead and write
Platform, DevOps, app, ETL and AI engineer groupsRead and writeNone
sa-acme-secrets-syncer (the workflows)Decrypt onlyDecrypt only

A developer with write access to the repository still cannot read or change a production secret, because Cloud KMS refuses the decrypt. This is least privilege applied at the key.

Design choices​

  • Git is the record. Every change to a secret is a reviewed commit with an author, a date and an approver. Nobody edits a value in the console.
  • One key per stage. Splitting by stage turns "who may see production secrets" into an IAM binding on a key, checked on every decrypt.
  • Secret Manager is what workloads read. Applications never read the repository; each stage project holds only its own stage's values.
  • No long-lived credentials. Both workflows exchange a GitHub token for a short-lived access token of the syncer; no key for the account exists, and a deny policy stops anyone from creating one.
  • Changes only. A secret whose latest version already holds the value is left alone, so a sync adds a version only when a value changed.

Where to go next​

What you receive lists the repository contents and the Baseline pieces the blueprint relies on. System architecture goes one level deeper.