Service inventory
The Data and ETL Blueprint deploys nine Azure services and the Unity Catalog objects of Azure Databricks into each stage, and relies on four more from the Baseline. Everything below is created by the blueprint's own code unless the table says it comes from the Baseline. Names use the stage prefix acme-eus2-<slug>, with the slugs dev, stg and prd.
How the pieces group
Azure services
| Service | Resource per stage | What it does in the blueprint |
|---|---|---|
| Data Lake Storage | stacmeeus2<slug>datalake | The lake: hierarchical namespace, ZRS, four containers, 30-day soft delete, tiering to cool after 90 days and archive after 180, shared keys off |
| Storage queue | etl-trigger in the lake | Holds the BlobCreated events until the trigger reads them |
| Event Grid | System topic acme-eus2-<slug>-datalake-events, subscription raw-input-json | Sends every new raw/input/*.json file to the queue, with the topic's managed identity |
| Private Link | pe-stacmeeus2<slug>datalake-blob, -dfs, -queue | Private endpoints of the lake in the stage's snet-endpoints subnet |
| Azure Databricks | Workspace acme-eus2-<slug>-dbw, access connector acme-eus2-<slug>-dbac | The Premium workspace, injected into the stage network, and the identity Unity Catalog uses on the lake |
| Container Apps | Environment acme-eus2-<slug>-cae, job acme-eus2-<slug>-etl-trigger-caj | The event-driven trigger: 0.5 vCPU and 1 GiB per execution, 300-second timeout, up to three parallel executions, nothing while idle |
| Managed identities | id-acme-eus2-<slug>-etl-trigger | The trigger identity: reads the queue and the raw files, pulls the image, reads the job ID, calls Databricks; the job runs as it |
| Key Vault | Secrets etl-trigger--image-tag and etl--job-id in the stage's contract vault | The deployed trigger image tag and the Databricks job ID |
| Resource groups and locks | -data-lake-rg, -databricks-rg, -etl-trigger-rg and the two managed groups | One resource group per unit; CanNotDelete locks on the lake and workspace groups in prod |
Every resource with logs sends them to the stage's application Log Analytics workspace through a diagnostic setting: storage reads, writes and deletes on blob and queue, the workspace's accounts, clusters, jobs, notebook, unityCatalog and workspace categories, and the Container Apps console and system logs.
Databricks and Unity Catalog objects
| Object | Name per stage | What it does |
|---|---|---|
| Storage credential | acme-eus2-<slug>-lake | The access connector's managed identity, bound to the stage's workspace |
| External locations | acme-eus2-<slug>-raw, -processed, -curated, -managed | One per container; path access with READ_FILES and WRITE_FILES |
| Catalog | acme_<slug> | Managed storage in the managed container, bound to the stage's workspace |
| Schemas | raw, processed, curated | One per zone, inheriting the catalog's managed location |
| Volume | /Volumes/acme_<slug>/processed/scripts | External volume on processed/scripts/, where the job reads etl_transform.py |
| Service principal | acme-eus2-<slug>-etl-trigger | The trigger identity registered in the workspace |
| Job | acme-eus2-<slug>-etl | The PySpark task, on a job cluster of Standard_D4ds_v5 nodes (two workers) per run by default |
| Grants | Catalog, external locations and job permissions | What the trigger identity and the two ETL engineer groups may do |
Used from the Baseline
| Service | Where it lives | Use |
|---|---|---|
| Virtual Network | Each stage subscription | The spoke and its subnets snet-endpoints, snet-dbx-host, snet-dbx-container and snet-aca |
| Private DNS | acme-core-network | The blob, dfs and queue Private Link zones that resolve the lake's private endpoints |
| Container Registry | acme-core-artifacts | The acme-azure-blueprint-etl-trigger repository in cracmeeus2artifacts, with locked tags |
| GitHub OIDC | Federated deployer identities | Short-lived Entra tokens for every workflow, with no stored secrets |
The Baseline also holds the stage's contract vault and Log Analytics workspace, the hub firewall or the spoke NAT gateway that carries cluster egress, and the regional Unity Catalog metastore root in acme-core-artifacts. The landing zone contract lists every value the blueprint reads.
Defaults per stage
The three stages start with the same sizing. Prod differs in two values: its job clusters use on-demand workers only, and its lake and workspace resource groups carry delete locks. The sizing per stage page lists every value and its price comment.