EFK with Fluentd sidecar and Deamonset

Here I will go through creating EFK stack with Fluentd working as side car and working as deamonset as well

We will start by creating a statefull set for elasticsearch

Shortcut: To launch the entire EFK in GKE use Google market placeone click deploy. It will launch Fluentd as a deamonset only and we can jump to Creating Fluentd as sidecar in this story

The following yaml will create elasticsearch statefullset save it and apply

Note: I am creating every thing in kube-logging namespace you can use different namespace if you like.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  generation: 1
  name: es-cluster
  namespace: kube-logging
spec:
  replicas: 2
  selector:
    matchLabels:
      app: elasticsearch
  serviceName: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: discovery.seed_hosts
          value: es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
        - name: cluster.initial_master_nodes
          value: es-cluster-0,es-cluster-1,es-cluster-2
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 100m
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        image: busybox
        imagePullPolicy: Always
        name: fix-permissions
        resources: {}
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: busybox
        imagePullPolicy: Always
        name: increase-vm-max-map
        resources: {}
        securityContext:
          privileged: true
      - command:
        - sh
        - -c
        - ulimit -n 65536
        image: busybox
        imagePullPolicy: Always
        name: increase-fd-ulimit
        resources: {}
        securityContext:
          privileged: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  volumeClaimTemplates:
  - metadata:
      labels:
        app: elasticsearch
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      storageClassName: $StorageClassName
      volumeMode: Filesystem

Now creating a headless service for elasticsearch

apiVersion: v1
kind: Service
metadata:
 labels:
    app: elasticsearch
  name: elasticsearch
  namespace: kube-logging
spec:
  clusterIP: None
  ports:
  - name: rest
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: inter-node
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch
  type: ClusterIP

Now we will create Kibana deployment and service

Here is the yaml for kibana deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kibana
  name: kibana
  namespace: kube-logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
      labels:
        app: kibana
    spec:
      containers:
      - env:
        - name: ELASTICSEARCH_URL
          value: http://elasticsearch:9200
        image: docker.elastic.co/kibana/kibana:7.2.0
        imagePullPolicy: IfNotPresent
        name: kibana
        ports:
        - containerPort: 5601
          protocol: TCP
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 100m

Here is the yaml file for kibana service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kibana
  name: kibana
  namespace: kube-logging
spec:
  ports:
  - nodePort: 31623
    port: 5601
    protocol: TCP
    targetPort: 5601
  selector:
    app: kibana
  type: LoadBalancer

Now we create fluentd as deamonset

We need to create kubernetes service account, a cluster role, a cluster role binding and the fluentd deamonset for it work. As a deamonset fluentd collects all logs from all pods stderr and stdout. We will use this later on in our fluentd sidecar.

Here is a yaml for creating all those mentioned above

apiVersion: v1
kind: ServiceAccount #creating service account
metadata:
  name: fluentd
  namespace: kube-logging
  labels:
    app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole #creating cluster role
metadata:
  name: fluentd
  labels:
    app: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding #creating Role binding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-logging
---
apiVersion: apps/v1 #creating Deamonset
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch.kube-logging.svc.cluster.local"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Now Creating Fluentd sidecar

It used where we use applications that are not used as primary processors and the logs of these applications are not stored in stderr and stdout of the containers. For these will create fluent as a sidecar to in our pod. Here will also create a configmap to store our fluentd.conf file and we will set the logs as stderr and stdout.

To do this we will add one more container in the deployment file. Here is the code snippet of the fluentd (this for an apache tomcat application). We are also creating a emptydir volume here which we are mounting in both the fluentd container and tomcat container with log directory.

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: tomcat
  namespace: test
spec:
  selector:
    matchLabels:
      app: tomcat  replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        team: app
        app: tomcat
    spec:
      containers:
      - name: test
        image: tomcat
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: logging-vol
          mountPath: /opt/apache-tomcat-7.0.57/logs
      - image: fluent/fluentd:v1.3-debian-1
        name: fluentd
        env:
        - name: FLUENT_UID
          value: root
        volumeMounts:
        - name: logging-vol
          mountPath: /opt/apache-tomcat-7.0.57/logs
        - name: log-config
          mountPath: /fluentd/etc/
      volumes:
          - name: logging-vol
            emptyDir: {}
          - name: log-config
            configMap:
              name: tomcat-config

We also need to create a configmap to store our fluentd config

apiVersion: v1
data:
  fluent.conf: |
    <source>
      @type tail
      format multiline
      format_firstline /[0-9]{2}-[A-Za-z]{3}-[0-9]{4}/
      format1 /^(?<datetime>[0-9]{2}-[A-Za-z]{3}-[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}) (?<Log-Level>[A-Z]*) (?<message>.*)$/
      path /opt/apache-tomcat-7.0.57/logs/catalina*,/opt/apache-tomcat-7.0.57/logs/manager*,/opt/apache-tomcat-7.0.57/logs/localhost.*,/opt/apache-tomcat-7.0.57/logs/host-manager*,/opt/apache-tomcat-7.0.57/logs/localhost_access_log*
      tag tomcat.logs   
    </source>
    <match tomcat.logs>
      @type stdout
    </match>
kind: ConfigMap
metadata:
  name: tomcat-config
  namespace: test

With this we will push logs to stdout from where it will be picked up by deamonset and pushed to elasticsearch and sent to kibana