Collecting Kubernetes Logs

The easiest way for applications running in a Kubernetes cluster to log to Seq is to use a native logging library and HTTP ingestion with the Seq instance's cluster internal DNS record, like http://seq.logging.svc.cluster.local.

A standard way to ingest logs in Kubernetes and other container orchestrators is to log directly to an output stream and let the ambient infrastructure collect the data and ship it off to the log server. This can be done using Fluent Bit to collect container logs and forward them to a Seq instance deployed from the Helm chart using GELF as the protocol.

Enabling GELF

The Seq Helm chart supports ingestion via GELF out-of-the-box. It can be enabled on deployment:

helm install -f config.yaml my-seq stable/seq

# config.yaml
gelf:
  enabled: true

Using Fluent Bit to collect logs

With Seq ready to ingest GELF events, the next step is to deploy Fluent Bit. It will be responsible for watching container logs on each node in the cluster and forwarding them as GELF events to Seq. Fluent Bit will take care of buffering for us in case Seq becomes unavailable.

See the Fluent Bit Kubernetes docs for more details.

Setting up a service account

First, create a namespace and a service account for Fluent Bit to use:

kubectl create namespace logging

kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

Creating a config map

Next, Fluent Bit needs some configuration to tell it where to find logs and how to filter and output them. This will be stored in a config map:

kubectl apply -f config.yaml

# config.yaml
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
apiVersion: v1
data:
    fluent-bit.conf: |
      [SERVICE]
          Flush         1
          Log_Level     info
          Daemon        off
          Parsers_File  parsers.conf
      @INCLUDE input-kubernetes.conf
      @INCLUDE filter-kubernetes.conf
      @INCLUDE output-gelf.conf
    input-kubernetes.conf: |
      [INPUT]
          Name              tail
          Tag               kube.*
          Path              /var/log/containers/*.log
          Parser            docker
          DB                /var/log/flb_kube.db
          Mem_Buf_Limit     5MB
          Skip_Long_Lines   On
          Refresh_Interval  10
    filter-kubernetes.conf: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Kube_URL            https://kubernetes.default.svc:443
          Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
          Kube_Tag_Prefix     kube.var.log.containers.
          Merge_Log           On
          Merge_Log_Key       log_processed
          K8S-Logging.Parser  On
          K8S-Logging.Exclude Off
    output-gelf.conf: |
      [OUTPUT]
          Name                    gelf
          Match                   kube.*
          Host                    ${FLUENT_GELF_HOST}
          Port                    ${FLUENT_GELF_PORT}
          Mode                    ${FLUENT_GELF_PROTOCOL}
          Gelf_Short_Message_Key  log
    parsers.conf: |
      [PARSER]
          Name   apache
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   apache2
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   apache_error
          Format regex
          Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
      [PARSER]
          Name   nginx
          Format regex
          Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   json
          Format json
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name        docker
          Format      json
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On
      [PARSER]
          Name        syslog
          Format      regex
          Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
          Time_Key    time
          Time_Format %b %d %H:%M:%S

See the Fluent Bit docs for more details on how its configuration is structured.

Deploy Fluent Bit

Fluent Bit is deployed as a daemon set to make sure it gets run on every node in the cluster:

kubectl apply -f daemon.yaml

# daemon.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.3.11
        imagePullPolicy: Always
        env:
        - name: FLUENT_GELF_HOST
          value: "seq.logging.svc.cluster.local"
        - name: FLUENT_GELF_PORT
          value: "12201"
        - name: FLUENT_GELF_PROTOCOL
          value: "tcp"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

Note the environment variables:

FLUENT_GELF_HOST: The internal domain your Seq instance is reachable on. This example uses seq.logging.svc.cluster.local.
FLUENT_GELF_PORT: The port GELF is listening on in the Seq instance. The default is 12201.
FLUENT_GELF_PROTOCOL: The protocol GELF is using. The default is TCP.

Ingest events

With Fluent Bit configured, anything an application writes to its output streams will be logged as an event in Seq.