Azure Data Lake Storage
Azure Data Lake Storage Gen2 (ADLS Gen2) is Azure Blob Storage with a hierarchical namespace: real directories, atomic renames and POSIX-style access control, the storage layout analytics engines expect. The Azure Data and ETL Blueprint keeps each stage's lake in one ADLS Gen2 account, and the Azure Enterprise Baseline uses one more for the Unity Catalog metastore root.
What it does
Turning on the hierarchical namespace makes a storage account a data lake: paths are directories instead of name prefixes, and the dfs endpoint serves the Data Lake API that Spark and Databricks use, beside the blob and queue endpoints. Everything else of a storage account applies: containers, Entra ID authorization, redundancy, soft delete, lifecycle tiering and private endpoints. Blob versioning is not available with a hierarchical namespace, so soft delete is the recovery path. A storage account can raise Blob Storage events through an Event Grid system topic, for example when a blob is created.
How BuiltForProd uses it
The data-lake module of the Data and ETL Blueprint creates stacmeeus2<stage>datalake per stage (24 characters at most) with four containers:
| Container | Zone |
|---|---|
raw | Input data as it arrives; new files under raw/input/ start the pipeline |
processed | Transformed data, and the ETL script under processed/scripts/ |
curated | Published data |
managed | Unity Catalog managed tables |
Settings. ZRS (GZRS or RAGZRS as the stage option), HTTPS only, TLS 1.2, infrastructure encryption, shared keys off and OAuth by default, so there is no account key or SAS token and all access is Azure RBAC and Unity Catalog grants. Blob and container soft delete keep deleted data for 30 days. A lifecycle rule moves raw/, processed/output/ and curated/ to cool after 90 days and archive after 180. Blob and queue read, write and delete logs go to the stage application workspace in Log Analytics. In prod, delete_locks puts a CanNotDelete lock on the lake's resource group.
Network. Three private endpoints, blob, dfs and queue, sit in the spoke's snet-endpoints with their records in the central Private Link zones. The public endpoint denies by default and admits only the two callers that cannot use a private endpoint: Event Grid, as a trusted Azure service, and the Unity Catalog access connector, as a named resource instance.
Events. The lake's Event Grid system topic delivers BlobCreated events for .json files under raw/input/ to the storage queue etl-trigger in the same account, through its own managed identity with Storage Queue Data Message Sender, and keeps each message for one day. A filter on the upload API passes only complete files, because a Data Lake API upload raises the event twice. The trigger job scales on that queue.
Identities. The access connector <prefix>-dbac is the system-assigned identity Unity Catalog uses for this lake: Storage Blob Data Contributor and Storage Queue Data Contributor on the account. The ETL code repository's identity may write only to the processed container, to upload its script.
The metastore root. The Baseline's uc-metastore unit creates stacmeeus2ucmetastore in acme-core-artifacts: an ADLS Gen2 account with the container metastore, ZRS, infrastructure encryption, Entra-only access and 30-day soft delete, with its own access connector. Every stage workspace of the blueprint attaches to the regional metastore built on it.
Terms you will see
| Term | Meaning |
|---|---|
| Hierarchical namespace | The storage account feature that makes it ADLS Gen2. |
dfs endpoint | The Data Lake API endpoint that Spark and Databricks use. |
| Zone | A container for one stage of the data's life: raw, processed, curated, managed. |
| Access connector | The managed identity Unity Catalog uses to reach the lake. |
| System topic | The Event Grid source that publishes a storage account's events. |
| Resource instance rule | A firewall exception for one named Azure resource. |
Where to read more
- Azure Data and ETL Blueprint overview for the pipeline from upload to curated data.
- Azure Storage for the controls every account shares.
- Private Link for the lake's private endpoints.