What you receive

You receive a complete data engineering environment, a working ETL pipeline and a governed data lake, deployed into your platform stage subscriptions on top of the Azure Enterprise Baseline, together with the two Git repositories that define it, customized for your organization, and the documentation to run and change it. After handover your engineers own all of it.

How delivery works

The blueprint is sold with the Baseline in Full Deployment with Blueprints, or on its own as a Blueprint Deployment for customers who already run the Baseline. In both cases the BuiltForProd team deploys it. See blueprints for how the blueprints fit together and purchasing and licensing for the terms.

The repositories

Repository	You receive
acme-azure-blueprint-etl-infra	Five OpenTofu modules, five Terragrunt units, one stack template, a stack file per stage, four guard scripts, pre-commit hooks, and plan, apply and drift-detection workflows
acme-azure-blueprint-etl-code	The PySpark transform, the Python trigger and its Dockerfile, unit tests for both, pre-commit hooks, three GitHub Actions workflows and the composite action that deploys a stage

acme in the names is your namespace. Your tenant, your stage subscriptions and your GitHub organization are set in the code before deployment. Every value a deployment must set carries a TODO: comment, and every switch you may change carries an @optional: comment, with its list price where it has one:

git grep -n "TODO: "
git grep -n "@optional: "

The running pipeline

In each of dev, staging and prod:

• One ADLS Gen2 storage account with the containers raw, processed, curated and managed, zone-redundant, Entra-only, with 30-day soft delete, tiering to cool and archive, and private endpoints for blob, Data Lake (dfs) and queue.
• An Event Grid system topic on the lake that sends every new raw/input/*.json file to the Storage queue etl-trigger.
• One Azure Databricks Premium workspace, injected into the stage's Databricks subnets, with secure cluster connectivity and diagnostics in the stage's Log Analytics workspace.
• Unity Catalog objects: a storage credential, four external locations, the catalog acme_<slug> with three schemas, the scripts volume and the grants for the trigger identity and the two ETL engineer groups.
• The Databricks job acme-eus2-<slug>-etl, which runs the PySpark transform on a job cluster as the trigger's identity.
• A Container Apps environment and the event-driven job acme-eus2-<slug>-etl-trigger-caj, which scales from zero on the queue and starts the Databricks job.

Upload a JSON file under raw/input/ and the pipeline runs end to end. The Delta table output is one stage value away: use_catalog = true writes acme_<slug>.processed.etl_output instead of Parquet files.

The delivery pipeline

• Pull requests to the code repository run lint, unit tests (the transform on a local Spark session, the trigger with mocked clients), a container build and an image vulnerability scan.
• A merge to main builds the trigger image once, locks its tag in the registry, uploads the transform script to dev, points the dev trigger job at the new image and runs one smoke-test execution.
• A GitHub Release adds the version tag to the same image and deploys it and the script to staging and, after a reviewer approves, to prod.
• Pull requests to the infrastructure repository plan all three stages; a merge applies them in order; a daily run reports drift.

The documentation

This documentation set: the architecture, the data lake, Azure Databricks, Unity Catalog governance, the event-driven trigger, the ETL code, the pipelines, querying, security, operations, performance tuning, how-to guides for day-2 work such as granting data access and releasing, and a reference of every unit, module, stack value and environment variable.

What stays with the Baseline

The subscriptions, networks, identities, the contract vault of each stage, the container registry and the regional Unity Catalog metastore root come from the Azure Enterprise Baseline. The blueprint reads what it needs from the Baseline and does not change it. The architecture overview shows how the two connect.