Skip to main content

Observable

An Observable system lets operators see what it is doing: every API call, every network flow, every metric, log line and trace, in one place, with alarms on the events that matter most. In the AWS Baseline the evidence goes to the audit account, the telemetry goes to the security account, and neither can be switched off from a workload account.

Where telemetry flows

The audit trail

  • One organization CloudTrail, multi-region, records management events (read and write) for every account, including accounts created later, plus S3 data events for the audit bucket itself. Log file validation is on, so tampering is detectable. The trail is KMS-encrypted and delivered both to the audit bucket and to a CloudWatch log group retained for 365 days.
  • Every VPC (the hub, the runner VPC and the four spokes) sends Flow Logs for all traffic at 60-second aggregation to the audit bucket.
  • AWS Config records every supported resource type in every account and region and delivers to the audit bucket every six hours; the organization aggregator in the audit account gives one view.
  • The audit bucket is versioned, KMS-encrypted with a dedicated key, TLS-only, and its current objects are never deleted: Standard-IA after 90 days, Glacier after 365. The acme-audit-protection SCP stops anyone but the root user from stopping CloudTrail or Config.

The CloudTrail explainer covers the trail; the Compliant page covers how the same evidence serves an audit.

Cross-account observability

CloudWatch Observability Access Manager (OAM) makes the security account the monitoring account. It holds the sink; the management, audit and network accounts and every workload account hold links that share metrics, log groups and X-Ray traces. Operators and the security team query every account from one console without holding a role in each. OAM is regional, so each region's links point to that region's sink. The CloudWatch OAM explainer describes the sink policy.

Alarms

Seven CIS metric filters on the CloudTrail log group in the management account watch for root account use, unauthorized API calls, IAM policy changes, CloudTrail configuration changes, S3 bucket policy changes, network changes and similar high-risk activity. The alarms live in the security account and read the management account's metrics through the OAM link, firing on one or more matches in five minutes to the acme-security-alerts SNS topic. The same topic receives GuardDuty findings of severity 7 and above, selected Security Hub findings and Config non-compliance. The subscriber email is security_alert_email in security.hcl; empty means no email subscription, and the topic still exists for other subscribers.

Application telemetry

SignalMechanismRetention
Container logsFluent Bit DaemonSet ships to CloudWatch with multiline parsing and Kubernetes metadata, health-check noise filtered, to /acme/<stage>/<app>/eks/<cluster>/applicationdev 30 days, staging 90, prod 365
API logsStructured JSON with an X-Request-ID correlation idAs above
Health/health liveness; /ready checks DocumentDB and RedisLive
EKS control planeAll five log types365 days
LambdaCloudWatch logs and X-Ray active tracing365 days
GlueContinuous CloudWatch logging and job metricsCloudWatch default
Client VPNConnection logs365 days
Public DNSRoute 53 query logs for the apex, prod and staging zones365 days
Network Firewall (when on)Alert and flow logs365 days

The Fluent Bit and X-Ray explainers describe the two shipping paths.

Drift as a signal

Observability includes the infrastructure itself. drift-detection.yml plans the network, identity and the three active workload accounts every weekday night and all fourteen accounts on Monday, and opens a GitHub issue labeled drift with the plan output when reality differs from the code. The drift page explains what to do with such an issue.

How you verify it

  • CloudTrail console in the management account: one organization trail, multi-region, log file validation enabled, delivering to the audit bucket.
  • S3 console in the audit account: the audit bucket with CloudTrail, Config and Flow Log prefixes, versioning on, lifecycle rules as above.
  • CloudWatch console in the security account, Settings, Monitoring account configuration: the sink and its linked source accounts.
  • CloudWatch Alarms in the security account: seven alarms in the acme/CISBenchmark namespace pointing at the security alerts topic.
  • The GitHub issues list of your landing-zone repository filtered by the drift label.

The CloudWatch explainer covers the metrics and log groups these consoles show.