Service inventory
The Data and ETL Blueprint deploys eleven AWS services into each stage and relies on two more in the Baseline's shared accounts. Everything below is created by the blueprint's own code unless the table says it comes from the Baseline.
How the pieces group
AWS services
| Service | What it does in the blueprint |
|---|---|
| S3 | Three buckets per stage, one per data lake zone, encrypted, versioned, private, with a lifecycle rule |
| Glue | Three Data Catalog databases, the PySpark ETL job and a crawler |
| Lake Formation | Administrators, registered bucket locations, LF-Tags and the grants that decide who reads which database and table |
| Lambda | The trigger that starts the Glue job when a JSON file lands in the raw zone, packaged as a container image |
| IAM | The Glue job role, the Lambda execution role and the Lake Formation service role |
| STS | Role assumption for the pipelines, from the automation account into each stage account |
| SSM Parameter Store | Reads the network values the Baseline publishes; holds the Lambda image tag each stage runs |
| KMS | Server-side encryption of every data lake object |
| EC2 | The security group that places the Lambda function in the stage's private subnets |
| CloudWatch Logs | The Lambda log group with 365-day retention and the Glue job's continuous logs and metrics |
| X-Ray | Active tracing of every sampled Lambda invocation |
Used from the Baseline
| Service | Where it lives | Use |
|---|---|---|
| ECR | Artifacts account | The Lambda image repository, with immutable tags |
| GitHub OIDC | Automation account | Short-lived AWS credentials for every workflow, with no stored keys |
The Baseline also provides the stage's VPC and its private subnets, and shares the Lambda function's traces and logs with the security account through CloudWatch cross-account observability.
Delivery tooling
The infrastructure is written with OpenTofu and Terragrunt. The Lambda function is packaged as a Docker image on the AWS Lambda Python base image, and every pipeline runs on GitHub Actions. Checkov and Trivy scan the infrastructure code in pre-commit hooks, Ruff lints the Python code, and Trivy scans every Lambda image before a pull request can merge.
The exact versions of all of these are on the versions page. The full catalog of services across BuiltForProd products is in AWS services used.