Is there a way to syslog from container to underlying k8s node? - docker

Question

I want to syslog from a container to the host Node -
Targeting fluentd (#127.0.0.1:5140) which runs on the node - https://docs.fluentd.org/input/syslog
e.g syslog from hello-server to the node (which hosts all of these namespaces)
I want to syslog output from hello-server container to fluentd running on node (#127.0.0.1:5140).
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default hello-server-7d8589854c-r4xfr 1/1 Running 0 21h
kube-system event-exporter-v0.2.4-5f7d5d7dd4-lgzg5 2/2 Running 0 6d6h
kube-system fluentd-gcp-scaler-7b895cbc89-bnb4z 1/1 Running 0 6d6h
kube-system fluentd-gcp-v3.2.0-4qcbs 2/2 Running 0 6d6h
kube-system fluentd-gcp-v3.2.0-jxnbn 2/2 Running 0 6d6h
kube-system fluentd-gcp-v3.2.0-k58x6 2/2 Running 0 6d6h
kube-system heapster-v1.6.0-beta.1-7778b45899-t8rz9 3/3 Running 0 6d6h
kube-system kube-dns-autoscaler-76fcd5f658-7hkgn 1/1 Running 0 6d6h
kube-system kube-dns-b46cc9485-279ws 4/4 Running 0 6d6h
kube-system kube-dns-b46cc9485-fbrm2 4/4 Running 0 6d6h
kube-system kube-proxy-gke-test-default-pool-040c0485-7zzj 1/1 Running 0 6d6h
kube-system kube-proxy-gke-test-default-pool-040c0485-ln02 1/1 Running 0 6d6h
kube-system kube-proxy-gke-test-default-pool-040c0485-w6kq 1/1 Running 0 6d6h
kube-system l7-default-backend-6f8697844f-bxn4z 1/1 Running 0 6d6h
kube-system metrics-server-v0.3.1-5b4d6d8d98-k7tz9 2/2 Running 0 6d6h
kube-system prometheus-to-sd-2g7jc 1/1 Running 0 6d6h
kube-system prometheus-to-sd-dck2n 1/1 Running 0 6d6h
kube-system prometheus-to-sd-hsc69 1/1 Running 0 6d6h
For some reason k8s does not allow us to use the built in syslog driver docker run --log-driver syslog.
Also, k8s does not allow me to connect with the underlying host using --network="host"
Has anyone tried anything similar? Maybe it would be easier to syslog remotely rather than trying to use the underlying syslog running on every node?

Answer 1

What you are actually looking at is the Stackdriver Logging Agent. According to the documentation at https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/#prerequisites:
If you’re using GKE and Stackdriver Logging is enabled in your cluster, you cannot change its configuration, because it’s managed and supported by GKE. However, you can disable the default integration and deploy your own.
The documentation then gives an example of rinning your own fluentd DaemonSet with custom ConfigMap. You'd need to run your own fluentd so you could configure a syslog input per https://docs.fluentd.org/input/syslog.
Then, since the fluentd is running as a DaemonSet, you would configure a Service to expose it to other pods and allow then to connect to it. If you are running the official upstream DaemonSet from https://github.com/fluent/fluentd-kubernetes-daemonset then a service might look like:
apiVersion: v1
kind: Service
namespace: kube-system
metadata:
name: fluentd
spec:
selector:
k8s-app: fluentd-logging
ports:
- protocol: UDP
port: 5140
targetPort: 5140
Then your applications can log to fluentd.kube-system:5140 (see using DNS at https://kubernetes.io/docs/concepts/services-networking/service/#dns).