Architecture overview
The Data and ETL Blueprint runs an event-driven data pipeline in each plat account of the AWS Baseline. A JSON file lands in the raw zone of an S3 data lake, a Lambda function starts a Glue PySpark job, and the job writes cleaned, partitioned Parquet to the processed zone. Lake Formation decides who may read which database, and a bucket policy blocks direct S3 access around it.
How it works
The pipeline has no always-on compute. Nothing runs between uploads: the upload fires an S3 event, the event invokes the function, and the function starts one job run over the files that arrived. The job flattens nested fields, parses timestamps, removes duplicate records, adds processing columns and writes the result partitioned by year, month and day. A crawler registers the result in the catalog.
Three zones
| Zone | Holds | Catalog database |
|---|---|---|
| Raw | JSON files as they arrive from source systems | acme-usw2-<stage>-raw-db |
| Processed | Transformed data in Parquet, deduplicated and enriched | acme-usw2-<stage>-processed-db |
| Curated | Business-ready data, aggregated and validated | acme-usw2-<stage>-curated-db |
Each zone is its own encrypted, versioned S3 bucket with public access blocked and a lifecycle rule that moves older objects to cheaper storage. Each has its own database in the Glue Data Catalog. Zones keep untrusted input apart from data other teams rely on.
Governed by Lake Formation
AWS Lake Formation governs access to the data. The buckets are registered with it, and access is granted per database and table:
- The Glue job reads the raw zone and reads and writes the processed zone.
- The Lambda function can describe the raw and processed databases, nothing more.
- Data engineers who sign in through IAM Identity Center can be granted read access to the processed and curated zones.
- Tags for zone, sensitivity and business domain are defined for tag-based access.
A bucket policy on every zone denies direct S3 access to anyone who is not an AWS service or one of a short list of platform roles, so a broad IAM policy cannot reach the files around Lake Formation.
Two repositories
| Repository | What it holds |
|---|---|
acme-aws-blueprint-etl-infra | The data lake, the catalog, the Glue job, the Lambda function and Lake Formation, as code |
acme-aws-blueprint-etl-code | The Glue transform script, the Lambda handler and its container image, the tests and pipelines |
Each stage (dev, staging and prod) is built from the same template of six Terragrunt units. The stages differ only in five values: Lambda memory and timeout, Glue worker type and count, and the switch that enforces Lake Formation access. All three start with the same values.
Delivery chain
The Lambda image is built once, on merge, and never rebuilt. A release adds a version tag to the same image, deploys it to staging and, after a reviewer approves, to prod. Each deployment also uploads the Glue script to the stage. The pattern is described in immutable artifacts and environments and promotion.
What it builds on
The blueprint creates no networks and no accounts. It reads each plat account's VPC and private subnets from the values the AWS Baseline publishes, runs the Lambda function in those private subnets, pulls its image from the registry in the artifacts account, and uses the Baseline's GitHub OIDC roles for every pipeline. The function's traces reach the security account through the Baseline's cross-account observability.
For the full list of services, see the service inventory. For what a customer receives, see what you receive.