Login
Start Trial

Pushing logs from the Kubernetes cluster using a fluentd

To easily send logs from the Kubernetes cluster one can use fluentd deamonset. It’s source code can be found at fluentd kubernetes deamonset GitHub page.

How does it work

The deamonset creates a logging agent pod on every node of the Kubernetes cluster and takes care of the logs collection and distribution.

How to run it

The following deamonset spec file can be used to send logs from the Kubernetes cluster.

Please remember to replace the CUSTOMER_TOKEN string with your token.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-logsense
  namespace: kube-system
data:
  fluent.conf: |
    @include kubernetes.conf

    <filter **>
      @type record_transformer
      <record>
        cs_customer_token CUSTOMER_TOKEN
      </record>
    </filter>

    <match **>
      @type forward
      send_timeout 60s
      recover_wait 10s
      hard_timeout 60s
      transport tls
      tls_verify_hostname true
      <server>
        host logs.logsense.com
        port 32714
      </server>
    </match>

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-logsense
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: fluentd-logsense
          image: fluent/fluentd-kubernetes-daemonset:v1.3-debian-gcs
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: config
              mountPath: /fluentd/etc/fluent.conf
              subPath: fluent.conf
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: config
          configMap:
            name: fluentd-logsense

RBAC rules

If your cluster uses RBAC Authorization then you need to use some additional configuration.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-logsense
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd-logsense
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - namespaces
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-logsense
roleRef:
  kind: ClusterRole
  name: fluentd-logsense
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: fluentd-logsense
    namespace: kube-system

Also you need to use those rules in the deamonset spec:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-logsense
  ...
spec:
  ...
  template:
    ...
    spec:
      serviceAccount: fluentd-logsense
      serviceAccountName: fluentd-logsense
      ...

Stay Connected

Subscribe to our newsletter.