Skip to main content

Fluent Bit

Fluent Bit is a small log processor. It tails container log files on a node, enriches each line with Kubernetes metadata and sends it to a destination. The Web App Blueprint runs the AWS build of it on every node and sends application logs to CloudWatch Logs.

What it does​

Deployed as a DaemonSet, Fluent Bit runs one pod per node and reads /var/log/containers/*.log. Filters parse JSON, merge multi-line entries and drop noise; an output plugin writes to CloudWatch, S3, Kinesis or others. It is the standard log shipper on EKS because it is light on CPU and memory.

How BuiltForProd uses it​

The fluent-bit unit in acme-aws-blueprint-webapp-infra installs the aws-for-fluent-bit chart (version 0.2.0) into the amazon-cloudwatch namespace, with tolerations so it also lands on tainted nodes and with requests and limits of 50m to 200m CPU and 128Mi to 256Mi memory.

  • Destination. One log group per stage, /acme/dev/blueprint-app/eks/acme-usw2-dev-eks/application, created by the module. Streams are named by the pod's app label with an app- prefix as fallback. Retention is a per-stage value: 30 days in dev, 90 in staging, 365 in prod.
  • Filters. The Kubernetes filter merges JSON log lines into structured fields; a multiline filter joins Python and Go stack traces into one event; a grep filter drops successful /health, /healthz and /livez requests and lines from the kube-probe user agent so the log group is not dominated by health checks (switchable with enable_healthcheck_filter).
  • Identity. This is the one add-on that uses IAM Roles for Service Accounts (IRSA) rather than Pod Identity: the role acme-usw2-dev-fluent-bit-irsa trusts the cluster's OIDC provider for the fluent-bit service account. The design decisions record why: Fluent Bit's C credential resolver does not read the Pod Identity token file. The policy allows writing streams and events to this log group only.
  • Application side. The API logs structured JSON with an X-Request-ID correlation id, so CloudWatch Logs Insights can filter by request across pods.

The AWS Baseline's CloudWatch OAM link shares every log group, this one included, with the security account, so the security team can query application logs without access to the platform account. EKS control-plane logs go to a different log group with 365-day retention, managed by the eks unit.

Terms you will see​

TermMeaning
DaemonSetA Kubernetes workload with exactly one pod on every node.
FilterA Fluent Bit stage that parses, merges or drops records.
Log streamA sequence of events inside a log group, here one per application.
IRSAIAM Roles for Service Accounts, the OIDC-based identity Fluent Bit uses.
RetentionHow long CloudWatch keeps events before deleting them.

Where to read more​