Skip to main content

Lambda

AWS Lambda runs code in response to events without a server to manage. In the Data and ETL Blueprint, one Lambda function is the trigger of the whole pipeline: a new JSON file in the raw bucket invokes it, and it starts the Glue job.

What it does​

A Lambda function is a piece of code, packaged as a zip file or a container image, that AWS runs when an event arrives. You set memory, timeout, environment variables and an execution role; AWS handles scaling and billing per invocation. A function can be attached to a VPC when it needs private network access.

How BuiltForProd uses it​

The lambda-trigger unit in acme-aws-blueprint-etl-infra creates acme-usw2-dev-etl-trigger through the platform's lambda module, a wrapper around terraform-aws-modules/lambda/aws 8.8.2. The sg-lambda unit provides its security group, which needs no ingress rules because the function is invoked, never called.

  • Packaging. The function runs the container image from the ECR repository acme-aws-blueprint-etl-lambda in the artifacts account. The image is built from src/functions/etl_trigger/Dockerfile in acme-aws-blueprint-etl-code on the AWS Lambda Python 3.12 base image.
  • Which tag. ECR tags are immutable, so the unit names no tag. The module owns the parameter /acme/usw2/dev/etl-trigger/image_tag, creates it with the value bootstrap and ignores later changes. The CD workflow writes every tag it deploys there, and the module reads it back at plan time, so a plan never rolls the function back.
  • Runtime settings. VPC-attached in the private subnets, X-Ray active tracing, 365-day log retention, and per-stage timeout (300 seconds) and memory_size (256 MB) from the stage stack file.
  • Permissions. The role may start and inspect Glue job runs and call lakeformation:GetDataAccess; it has no direct S3 data access.
  • Trigger. The glue-job module adds an S3 event notification on the raw bucket for s3:ObjectCreated:* under the input/ prefix, plus the invoke permission for the S3 service.

The handler in main.py reads the S3 records and keeps the .json keys, then calls start_job_run on the Glue job named by the GLUE_JOB_NAME variable the unit sets. Its arguments are those object keys, comma separated, so a run processes only what arrived rather than the whole input/ prefix and never appends the same rows twice; the raw and processed catalog database names; the run date; and USE_CATALOG, which the unit sets to false. The cd-integration.yml workflow updates the function code with aws lambda update-function-code, waits for the update, records the tag in Parameter Store and runs a smoke invocation with an empty record list.

The AWS Baseline also relies on Lambda indirectly: the optional self-hosted runner module in the automation account is built from Lambda functions that receive GitHub webhooks and scale EC2 runners.

Terms you will see​

TermMeaning
HandlerThe function entry point, here main.handler.
Package typeZip or Image; the trigger uses Image.
Execution roleThe IAM role the function runs as.
Event notificationThe S3 configuration that invokes the function on new objects.
Active tracingX-Ray instrumentation of every invocation.

Where to read more​