Logs

Here are some of the general challenges of logging in a microservices application:

• Understanding the end-to-end processing of a client request, where multiple services might be invoked to handle a single request.
• Consolidating logs from multiple services into a single aggregated view.
• Parsing logs that come from multiple sources, which use their own logging schemas or have no particular schema. Logs may be generated by third-party components that you don’t control.
• Microservices architectures often generate a larger volume of logs than traditional monoliths, because there are more services, network calls, and steps in a transaction. That means logging itself can be a performance or resource bottleneck for the application.

There are some additional challenges for a Kubernetes-based architecture:

• Containers can move around and be rescheduled.
• Kubernetes has a networking abstraction that uses virtual IP addresses and port mappings.

In Kubernetes, the standard approach to logging is for a container to write logs to stdout and stderr. The container engine redirects these streams to a logging driver, at the very least storing them individually on each executing node. For ease of querying, and to prevent possible loss of log data if a node crashes, the usual approach is to collect the logs from each node and send them to a central storage location.

Many organizations use Fluentd with Elasticsearch, running either within the Kubernetes cluster or externally hosted by a service provider. Fluentd is an open-source data collector and a Cloud Native Computing Foundation graduated project, and Elasticsearch is a document database that is for search.

A fast, lightweight, and highly scalable alternative to Fluentd is Fluent-Bit. Fluent-Bit is the preferred choice for cloud and containerized environments and is part of the Fluentd organization.

Both, Fluent-Bit and Elasticsearch, are already predefined in our Kubernetes cluster, the former using the Fluent Bit Helm Chart and the latter using the Elastic Cloud on Kubernetes (ECK) Helm Chart (ECK-Stack) .

# uses its own namespace "elastic-system"
kubectl create -f https://download.elastic.co/downloads/eck/2.16.1/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.16.1/operator.yaml

kubectl apply -f - <<.EOF
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: exercises
  namespace: logging
spec:
  version: 7.8.1
  nodeSets:
    - name: default
      count: 1
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
.EOF

until kubectl get elasticsearch exercises --namespace logging | grep --quiet Ready ; do sleep 1; date; done

kubectl apply -f - <<.EOF
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: exercises
  namespace: logging
spec:
  version: 7.8.1
  count: 1
  elasticsearchRef:
    name: exercises
.EOF

until kubectl get kibana exercises --namespace logging | grep --quiet green ; do sleep 1; date; done

ELASTICPASSWORD=$(kubectl get secret --namespace logging exercises-es-elastic-user --output go-template='{{.data.elastic | base64decode}}')
kubectl apply -f - <<.EOF
# https://medium.com/kubernetes-tutorials/cluster-level-logging-in-kubernetes-with-fluentd-e59aa2b6093a
# https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes
#
# this file assumes there is an Elasticsearch instance called "exercises", as created by the Elasticsearch Operator,
# running in the namespace "logging", while fluentd must run in the namespace "kube-system" to do its job
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  labels:
    k8s-app: fluentd-logging
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - namespaces
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1.11.2-debian-elasticsearch7-1.0
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "exercises-es-http.logging.svc.cluster.local"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"
            - name: FLUENT_ELASTICSEARCH_SCHEME
              value: "https"
            # XXX, cf. https://www.elastic.co/guide/en/cloud-on-k8s/1.2/k8s-tls-certificates.html
            - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
              value: "false"
            - name: FLUENTD_SYSTEMD_CONF
              value: disable
            - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
              value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
            - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
              value: '["/var/log/containers/fluentd-*"]'
            # X-Pack Authentication
            # =====================
            - name: FLUENT_ELASTICSEARCH_USER
              value: "elastic"
            - name: FLUENT_ELASTICSEARCH_PASSWORD
              value: "$ELASTICPASSWORD"
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
.EOF

kubectl patch svc exercises-kb-http -p '{"spec": {"ports": [{"port": 5601,"targetPort": 5601,"name": "kibana"}],"type": "LoadBalancer"}}' --namespace logging

until kubectl get svc --namespace logging exercises-kb-http -o jsonpath="{.status.loadBalancer.ingress[0].ip}" | grep -v --silent pending ; do sleep 1; date; done
IP=$(kubectl get svc --namespace logging exercises-kb-http -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
ELASTICPASSWORD=$(kubectl get secret --namespace logging exercises-es-elastic-user --output go-template='{{.data.elastic | base64decode}}')
curl --silent --user "elastic:$ELASTICPASSWORD" \
    "https://$IP:5601/api/saved_objects/index-pattern/exercises" \
    --header 'kbn-xsrf: true' --header 'Content-Type: application/json' \
    --data @- <<.EOF
{
  "attributes": {
    "title": "logstash-*",
    "timeFieldName": "@timestamp"
  }
}
.EOF

Exercise - Explore Elasticsearch

First let’s take a look which Custom Resource Definitions (CRD) the Operator provides:

kubectl get crd | grep -e '^NAME' -e 'elastic.co'

NAME                                                   CREATED AT
agents.agent.k8s.elastic.co                            2025-03-26T13:40:29Z
apmservers.apm.k8s.elastic.co                          2025-03-26T13:40:29Z
beats.beat.k8s.elastic.co                              2025-03-26T13:40:29Z
elasticmapsservers.maps.k8s.elastic.co                 2025-03-26T13:40:29Z
elasticsearchautoscalers.autoscaling.k8s.elastic.co    2025-03-26T13:40:29Z
elasticsearches.elasticsearch.k8s.elastic.co           2025-03-26T13:40:30Z
enterprisesearches.enterprisesearch.k8s.elastic.co     2025-03-26T13:40:29Z
kibanas.kibana.k8s.elastic.co                          2025-03-26T13:40:30Z
logstashes.logstash.k8s.elastic.co                     2025-03-26T13:40:29Z
stackconfigpolicies.stackconfigpolicy.k8s.elastic.co   2025-03-26T13:40:29Z

Of these we only use elasticsearches and kibanas here. What is the state of our elasticsearch?

kubectl get elasticsearch --selector app.kubernetes.io/name=eck-elasticsearch --namespace logging

NAME            HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch   yellow    1       8.17.0    Ready   60m

Huh, HEALTH is only yellow? Well, we installed a minimal Elasticsearch cluster consisting of only a single node (cf. NODES showing 1), so there cannot be any data redundancy, and hence the cluster health will never be green. But as long as it is not red and the PHASE remains Ready this should suffice.

Let’s first get the access credentials from a secret stored by the Operator (just like we had with the Postgres Operator already):

export ELASTICPASSWORD=$(kubectl get secret --namespace logging elasticsearch-es-elastic-user --output go-template='{{.data.elastic | base64decode}}')

With that we can query the Elasticsearch cluster, getting some general information:

curl --silent --user "elastic:$ELASTICPASSWORD" https://zwtj1.elasticsearch.cloudtrainings.online

{
  "name": "elasticsearch-es-default-0",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "kTeJp6oDTomSiKai-Kku8Q",
  "version": {
    "number": "8.17.0",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
    "build_date": "2024-12-11T12:08:05.663969764Z",
    "build_snapshot": false,
    "lucene_version": "9.12.0",
    "minimum_wire_compatibility_version": "7.17.0",
    "minimum_index_compatibility_version": "7.0.0"
  },
  "tagline": "You Know, for Search"
}

List Compact and aligned text (CAT) APIs :

curl --silent --user "elastic:$ELASTICPASSWORD" https://zwtj1.elasticsearch.cloudtrainings.online/_cat

/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/tasks
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/thread_pool/{thread_pools}
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}
/_cat/templates
/_cat/component_templates/_cat/ml/anomaly_detectors
/_cat/ml/anomaly_detectors/{job_id}
/_cat/ml/datafeeds
/_cat/ml/datafeeds/{datafeed_id}
/_cat/ml/trained_models
/_cat/ml/trained_models/{model_id}
/_cat/ml/data_frame/analytics
/_cat/ml/data_frame/analytics/{id}
/_cat/transforms
/_cat/transforms/{transform_id}

Now the cluster health:

curl --silent --user "elastic:$ELASTICPASSWORD" "https://zwtj1.elasticsearch.cloudtrainings.online/_cat/health?v"

epoch      timestamp cluster       status node.total node.data shards pri relo init unassign unassign.pri pending_tasks ... active_shards_percent
1743492716 07:31:56  elasticsearch yellow          1         1     33  33    0    0        2            0             0   -                 94.3%

And what indices for storing data already exist:

curl --silent --user "elastic:$ELASTICPASSWORD" "https://zwtj1.elasticsearch.cloudtrainings.online/_cat/indices?v" | grep -e '^health' -e logstash -e node

health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open   logstash-2025.04.01 Ml_gOw-tSBeyc8mL-tDWtA   1   1       2102            0      1.9mb          1.9mb        1.9mb
yellow open   node-2025.04.01     R_I9g2jPRzqykYTTjE5mDA   1   1         36            0    156.5kb        156.5kb      156.5kb

Exercise - Explore log data stored in Elasticsearch

Using the HTTP API we can also query what has been stored in our Elasticsearch cluster, for example what is today’s most recent log entry overall, yielding all data available (i.e. raw log data as well as parsed data):

curl --silent --user "elastic:$ELASTICPASSWORD" \
    "https://zwtj1.elasticsearch.cloudtrainings.online/logstash-$(date +%Y.%m.%d)/_search?pretty" \
    --header 'Content-Type: application/json' --data @- <<.EOF
{
  "query": {
    "match_all": {}
  },
  "sort": [
    {
      "@timestamp": "desc"
    }
  ],
  "from": 0,
  "size": 1
}
.EOF

Or what is today’s most recent log entry in our individual namespace, but let’s only display a subset of the result data:

curl --silent --user "elastic:$ELASTICPASSWORD" \
    "https://zwtj1.elasticsearch.cloudtrainings.online/logstash-$(date +%Y.%m.%d)/_search?pretty" \
    --header 'Content-Type: application/json' --data @- <<.EOF
{
  "query": {
    "match_phrase": {
      "kubernetes.namespace_name": "$NAMESPACE"
    }
  },
  "sort": [
    {
      "@timestamp": "desc"
    }
  ],
  "from": 0,
  "size": 10,
  "_source": [
    "@timestamp",
    "log",
    "stream",
    "kubernetes.pod_name",
    "kubernetes.container_name"
  ]
}
.EOF

which could yield something along the lines of

{
  "took": 38,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 284,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "logstash-2025.04.01",
        "_id": "smhP8JUBiV9UiUN_0Plq",
        "_score": null,
        "_source": {
          "@timestamp": "2025-04-01T07:45:16.700Z",
          "stream": "stdout",
          "log": "Hibernate: select t1_0.todo from todo t1_0",
          "kubernetes": {
            "pod_name": "todobackend-7d6f7bbf9c-tn47z",
            "container_name": "todobackend"
          }
        },
        "sort": [1743493516700]
      },
      {
        "...": "..."
      }
    ]
  }
}

Exercise - Generate and retrieve log data

Let’s generate some easy-to-digest log data within our cluster. In another terminal execute

kubectl run count --image busybox --command -- sh -c 'i=0; while sleep 2; do echo "$i: $(date)"; i=$((i+1)); done' && kubectl get pod count --watch

Once the Pod is running we can check when the fluent-bit Pod running on the corresponding node will start picking up its log file for forwarding to Elasticsearch:

kubectl logs --selector app.kubernetes.io/name=fluent-bit --namespace logging --tail 1000 --follow \
    | grep "count_${NAMESPACE}_count-$(kubectl get pod count -o jsonpath='{.status.containerStatuses[].containerID}' | cut -d'/' -f3)"

This might take up to 30 seconds for noticing there is a new file for forwarding, and then look like this:

[2025/04/01 14:32:40] [ info] [input:tail:tail.0] inotify_fs_add(): inode=42231 watch_fd=64 name=/var/log/containers/count_<your_namespace>_count-<your_container_id>.log

Kill the count Pod again by Ctrl+C on the watch terminal from above and then execute kubectl delete pod count.

Exercise - Access log data using Kibana

Of course, while querying log data via curl is convenient, and the HTTP API opens up the door for any subsequent log data processing, we humans might prefer a different interface for digesting our logs. What is the state of our Kibana in the cluster:

kubectl get kibana --selector app.kubernetes.io/name=eck-kibana --namespace logging

NAME                       HEALTH   NODES   VERSION   AGE
elastic-stack-eck-kibana   green    1       8.17.0    7h46m

There is an Ingress for accessing the Kibana Dashboard, execute

kubectl get ingresses --selector app.kubernetes.io/name=eck-kibana --namespace logging

to find the host to access Kibana through your local browser at https://zwtj1.kibana.cloudtrainings.online . Access Kibana with user: elastic and password: echo $ELASTICPASSWORD (see above).

If you see the Welcome page, just click on Explore on my own:

You should see the Home page like in the following picture:

If you see the Explore page, just click on Try ES|QL:

In Kibana you can query for logs with specific values. As you can remember the created Pod with the date output have the name count. So search this container by adding the query

FROM logs* | WHERE kubernetes.namespace_name == "<your_namespace>" AND kubernetes.container_name == "count" | SORT @timestamp DESC

to the Search field, replacing <your_namespace> and run it. If you do not get any results, change the date range for your search in the Date quick select:

In the log field you can see the date output from the executed container. Some other useful information is provided here by Kibana. Click on Toggle dialog with details to get detail information about the logs. Hide null fields if they are not already hidden. To navigate through the result details use the navigation bar Result |< < 1 of 223 > >|: