CloudWatch
Amazon CloudWatch collects logs and metrics from AWS services and applications, and raises alarms when a metric crosses a threshold. In the platform it holds every log stream that is not archived to S3, and it runs the security alarms.
What it does
CloudWatch Logs stores log lines in log groups, each with a retention period after which lines are deleted. A metric is a time series such as CPU use or a count of matching log lines. A metric filter turns log lines that match a pattern into a metric. An alarm watches a metric and changes state when a threshold is crossed, then notifies a target such as an SNS topic. Logs Insights queries log groups with a query language. CloudWatch is regional: a log group and its alarms live in one region.
How BuiltForProd uses it
The security alarms are the most important use. Seven CIS metric filters sit on the organization CloudTrail log group in core-root (units/cis-metric-filters, module modules/cloudwatch-alarms) and publish to the acme/CISBenchmark namespace. The matching alarms live in core-security (units/cis-alarms), fire when a metric reaches one in five minutes, and notify the acme-security-alerts SNS topic. Because the metrics belong to another account, each alarm uses a metric query with account_id set to core-root, evaluated through the OAM link.
Log groups and their retention:
| Log group | Source | Retention |
|---|---|---|
| CloudTrail log group | Organization trail, core-root | 365 days |
/aws/route53/<zone> | Public DNS query logs for the apex, prod and staging zones, us-east-1 | 365 days |
| Client VPN connection logs | The Client VPN endpoint in core-network, when enabled | 365 days |
| Network Firewall alert and flow logs | The hub firewall, when enabled | 365 days |
| EKS control-plane logs | All five log types of every cluster | 365 days |
| Lambda function logs | The ETL trigger function | 365 days |
/acme/<stage>/<app>/eks/<cluster>/application | Application logs shipped by Fluent Bit | dev 30, staging 90, prod 365 days |
The retention standard is at least one year for anything security or audit related, because that covers a SOC 2 audit period; application logs in lower environments are kept for less to save cost. Glue jobs write continuous logs and job metrics to CloudWatch as well.
Two more services feed CloudWatch. GuardDuty findings and Security Hub findings arrive through EventBridge rather than logs. X-Ray traces from the Lambda trigger are shared to core-security alongside logs and metrics.
The auditor permission sets can describe alarms and log groups but are denied logs:GetLogEvents and logs:StartQuery, because log content may contain personal data.
Terms you will see
| Term | Meaning |
|---|---|
| Log group | A named container of log streams with one retention setting. |
| Retention | How many days CloudWatch keeps a log group's lines before deleting them. |
| Metric filter | A pattern on a log group that emits a metric when lines match. |
| Namespace | A grouping of metrics, such as acme/CISBenchmark. |
| Alarm | A threshold on a metric that notifies a target when crossed. |
| Metric query | An alarm expression that can reference a metric in a linked account. |
| Logs Insights | The query interface for searching log groups. |
Where to read more
- AWS Baseline overview for the security account that owns the alarms.
- CloudWatch OAM for how the security account sees the telemetry of the management, audit, network and workload accounts.
- Fluent Bit for how application logs reach CloudWatch.