Observable
An Observable system lets operators see what it is doing: every API call, every network flow, every metric, log line and trace, in one place, with alarms on the events that matter most. In the AWS Baseline the evidence goes to the audit account, the telemetry goes to the security account, and neither can be switched off from a workload account.
Where telemetry flows
The audit trail
- One organization CloudTrail, multi-region, records management events (read and write) for every account, including accounts created later, plus S3 data events for the audit bucket itself. Log file validation is on, so tampering is detectable. The trail is KMS-encrypted and delivered both to the audit bucket and to a CloudWatch log group retained for 365 days.
- Every VPC (the hub, the runner VPC and the four spokes) sends Flow Logs for all traffic at 60-second aggregation to the audit bucket.
- AWS Config records every supported resource type in every account and region and delivers to the audit bucket every six hours; the organization aggregator in the
auditaccount gives one view. - The audit bucket is versioned, KMS-encrypted with a dedicated key, TLS-only, and its current objects are never deleted: Standard-IA after 90 days, Glacier after 365. The
acme-audit-protectionSCP stops anyone but the root user from stopping CloudTrail or Config.
The CloudTrail explainer covers the trail; the Compliant page covers how the same evidence serves an audit.
Cross-account observability
CloudWatch Observability Access Manager (OAM) makes the security account the monitoring account. It holds the sink; the management, audit and network accounts and every workload account hold links that share metrics, log groups and X-Ray traces. Operators and the security team query every account from one console without holding a role in each. OAM is regional, so each region's links point to that region's sink. The CloudWatch OAM explainer describes the sink policy.
Alarms
Seven CIS metric filters on the CloudTrail log group in the management account watch for root account use, unauthorized API calls, IAM policy changes, CloudTrail configuration changes, S3 bucket policy changes, network changes and similar high-risk activity. The alarms live in the security account and read the management account's metrics through the OAM link, firing on one or more matches in five minutes to the acme-security-alerts SNS topic. The same topic receives GuardDuty findings of severity 7 and above, selected Security Hub findings and Config non-compliance. The subscriber email is security_alert_email in security.hcl; empty means no email subscription, and the topic still exists for other subscribers.
Application telemetry
| Signal | Mechanism | Retention |
|---|---|---|
| Container logs | Fluent Bit DaemonSet ships to CloudWatch with multiline parsing and Kubernetes metadata, health-check noise filtered, to /acme/<stage>/<app>/eks/<cluster>/application | dev 30 days, staging 90, prod 365 |
| API logs | Structured JSON with an X-Request-ID correlation id | As above |
| Health | /health liveness; /ready checks DocumentDB and Redis | Live |
| EKS control plane | All five log types | 365 days |
| Lambda | CloudWatch logs and X-Ray active tracing | 365 days |
| Glue | Continuous CloudWatch logging and job metrics | CloudWatch default |
| Client VPN | Connection logs | 365 days |
| Public DNS | Route 53 query logs for the apex, prod and staging zones | 365 days |
| Network Firewall (when on) | Alert and flow logs | 365 days |
The Fluent Bit and X-Ray explainers describe the two shipping paths.
Drift as a signal
Observability includes the infrastructure itself. drift-detection.yml plans the network, identity and the three active workload accounts every weekday night and all fourteen accounts on Monday, and opens a GitHub issue labeled drift with the plan output when reality differs from the code. The drift page explains what to do with such an issue.
How you verify it
- CloudTrail console in the management account: one organization trail, multi-region, log file validation enabled, delivering to the audit bucket.
- S3 console in the
auditaccount: the audit bucket with CloudTrail, Config and Flow Log prefixes, versioning on, lifecycle rules as above. - CloudWatch console in the
securityaccount, Settings, Monitoring account configuration: the sink and its linked source accounts. - CloudWatch Alarms in the
securityaccount: seven alarms in theacme/CISBenchmarknamespace pointing at the security alerts topic. - The GitHub issues list of your landing-zone repository filtered by the
driftlabel.
The CloudWatch explainer covers the metrics and log groups these consoles show.