Skip to main content

Glue

AWS Glue is the serverless data integration service. It holds the Data Catalog (table definitions over files in S3), runs Spark jobs without a cluster to manage, and crawls storage to discover schemas. The Data and ETL Blueprint uses all three parts.

What it does​

The Data Catalog is a metastore: databases and tables that describe where data lives and what its columns are. Glue jobs run PySpark scripts on a pool of workers you size per job. Crawlers scan a location, infer a schema and write tables into the catalog. Athena, Lake Formation and Spark all read the same catalog.

How BuiltForProd uses it​

Two units in acme-aws-blueprint-etl-infra cover Glue, deployed once per stage.

glue-catalog creates one database per data lake zone: acme-usw2-dev-raw-db, acme-usw2-dev-processed-db and acme-usw2-dev-curated-db, each pointing at its bucket. The default IAM_ALLOWED_PRINCIPALS permission is removed on every database, so IAM policies alone cannot read tables; Lake Formation grants are required.

glue-etl creates the job acme-usw2-dev-etl and the crawler acme-usw2-dev-crawler through the glue-job module:

  • Glue 4.0, glueetl command, Python 3, script s3://<processed bucket>/scripts/etl_transform.py.
  • Worker type and count are per-stage values (G.1X, 2 workers by default) in the stage stack file. max_concurrent_runs defaults to 3, so files arriving together each get a run.
  • Continuous CloudWatch logging, job metrics and Data Catalog access are on; the output format is plain Parquet.
  • The job role acme-usw2-dev-glue-role holds the AWSGlueServiceRole policy, Lake Formation data-access actions, read on the script location and on both buckets, and write on the processed bucket. Spark writes the output and its temporary files with the role's own credentials, so read alone is not enough; --TempDir points at tmp/ in the same bucket.
  • The crawler scans output/ in the processed bucket into the processed database, so the tables it registers describe what the job wrote. It uses IAM credentials for schema discovery only; data access at query time still goes through Lake Formation.
  • An optional KMS key adds a Glue security configuration that encrypts logs and bookmarks.
  • The module also wires the S3 event notification that invokes the Lambda trigger.

The script lives in acme-aws-blueprint-etl-code at src/glue/etl_transform.py. It flattens the nested metadata fields, parses timestamps, adds processed_at and etl_version, derives year, month and day partition columns, removes duplicate ids and drops rows without required fields, then writes Parquet into the processed database. Where it reads from is a setting, --use_catalog, which the trigger passes as false: a run reads the JSON objects the S3 event named, and the crawler registers the output afterwards. Set it to true once a catalog table describes the raw input. Each CD workflow uploads the script to the stage's bucket with aws s3 cp.

Terms you will see​

TermMeaning
Data CatalogThe Glue metastore of databases and tables.
Job runOne execution of the ETL job, with its arguments and status.
Worker typeThe size of each Spark worker, such as G.1X.
CrawlerThe scanner that creates or updates tables from files in S3.
Job bookmarkGlue's record of which input it has already processed.
--TempDirThe S3 prefix Spark writes its intermediate files to.

Where to read more​