Architecture overview
The Data and ETL Blueprint runs an event-driven data pipeline in each platform stage subscription of the Azure Enterprise Baseline. A JSON file lands in the raw zone of an Azure Data Lake Storage (ADLS) Gen2 lake, an event wakes an Azure Container Apps job, the job starts an Azure Databricks job run, and the run writes cleaned, partitioned Parquet, or a Delta table governed by Unity Catalog. Every data plane is private and every identity authenticates with Microsoft Entra ID.
How it works
Nothing runs between uploads. The upload raises a BlobCreated event, the lake's Event Grid system topic delivers it to a Storage queue in the same lake, and the queue length makes KEDA start one execution of the Container Apps job. The execution reads up to 32 events, keeps the files that still exist, starts one run of the Databricks job for the batch and exits. The job flattens nested fields, parses timestamps, removes duplicate records, adds processing columns and appends the result partitioned by year, month and day.
The data flow page follows one file through the pipeline step by step.
One stage
Each stage (dev, staging and prod) gets the same five building blocks, from one template:
| Building block | What it is |
|---|---|
| Data lake | One ADLS Gen2 storage account with the zone containers raw, processed, curated and the Unity Catalog container managed, private endpoints only for its callers, Entra-only data plane |
| Workspace | One Azure Databricks Premium workspace injected into the stage network, with secure cluster connectivity: cluster nodes have no public IP address |
| Unity Catalog | A storage credential, one external location per container, the catalog acme_<slug> with the schemas raw, processed and curated, and a volume that holds the ETL script |
| Trigger | An Event Grid system topic, the queue etl-trigger and an event-driven Container Apps job that scales from zero |
| Job | The Databricks job acme-eus2-<slug>-etl: a PySpark task on a job cluster created per run, running as the trigger's identity |
The stage slugs are dev, stg and prd, so the dev lake is stacmeeus2devdatalake, the dev workspace acme-eus2-dev-dbw and the dev catalog acme_dev. The system architecture page places every resource in its resource group.
Governed by Unity Catalog
Unity Catalog decides who reads and writes the data. People have no Azure role on the lake's data plane. Databricks reaches the lake through one access connector, whose managed identity is the stage's storage credential, and Unity Catalog grants decide what each principal may do through it:
- The trigger identity, which the job runs as, reads
raw, reads and writesprocessed, and creates and modifies tables in the catalog. - The lead ETL engineers (
ACME_LeadETLEngineers) hold all privileges on the catalog and read and write files in the three zones. - The ETL engineers (
ACME_ETLEngineers) read the catalog's tables and volumes and read files incurated.
The lake's firewall denies every caller that is not on a private endpoint, except Event Grid (a trusted Azure service) and the access connector (an allowed resource instance).
Two repositories
| Repository | What it holds |
|---|---|
acme-azure-blueprint-etl-infra | The lake, the event path, the workspace, the Unity Catalog objects, the Databricks job and the Container Apps job, as Terragrunt code |
acme-azure-blueprint-etl-code | The PySpark transform, the trigger container, their tests and the pipelines that ship both |
The trigger image is built once on merge, locked in the Baseline's container registry and promoted unchanged from dev to staging and prod. Each deployment also uploads the transform script to the stage's lake. The pattern follows immutable artifacts and environments and promotion. See repositories.
What it builds on
The blueprint creates no network, no subscription and no registry. It reads the stage's virtual network, subnets, Private Link zones, Log Analytics workspace, registry and code deployer identity from the Baseline's contract vault in each stage subscription, and attaches its workspaces to the regional Unity Catalog metastore whose root storage the Baseline keeps in acme-core-artifacts. The reasoning behind this split is in the Baseline's data platform decision, and the values read are listed on the landing zone contract page.
For the full list of resources, see the service inventory. For what a customer receives, see what you receive.