How to push logs to fluentd deamonset using logrus - docker

Question

I have a simple golang application running inside kubernetes which just logs whatever text we place in request params. Code :
package main
import (
"github.com/sirupsen/logrus"
"net/http"
"os"
)
var (
logger = logrus.New()
)
func main() {
//logFile, err := os.OpenFile("/var/log/app.log", os.O_CREATE | os.O_WRONLY | os.O_APPEND, 0666)
//if err != nil {
// panic(err)
//}
logger.Out = os.Stdout
http.HandleFunc("/get", func(w http.ResponseWriter, r *http.Request) {
content := r.URL.Query().Get("m")
if content == "" {
logger.Warn("m is empty")
content = "nothing"
}
logger.Info("content is - ", content)
w.Write([]byte(content))
})
logger.Info("starting server")
http.ListenAndServe(":9999", nil)
}
I have fluentd deamonset running on my local which is connected to elasticsearch (also running in local but not in kubernetes). Now I want my logs to go to elasticsearch and ultimately appear in kibana but that's not happening and I am not able to debug.
My fluentd yaml looks like -
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluentd-logging
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "minikube-host"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENT_UID
value: "0"
# X-Pack Authentication
# =====================
- name: FLUENT_ELASTICSEARCH_USER
value: "elastic"
- name: FLUENT_ELASTICSEARCH_PASSWORD
value: "MyPw123"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Here minikube-host is just a proxy service that fluentd uses to connect to host machine (elasticsearch runs directly on host).
fluentd rbac looks like -
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
version: v1
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-system
fluentd is able to push logs to ES because I can see logs from fluentd pod in kibana. Actually I am able to see logs from fluentd pod only (no app logs, no logs from pods running in kube-system)
Not sure if it is relevant: my application runs inside docker with dockerfile
FROM golang:1.15 as base_image
WORKDIR /go/src/demo
ADD . .
RUN go get -d ./...
RUN go build -o /build/main-linux main.go
WORKDIR /build
CMD ["./main-linux"]
and my application deployment file is -
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
spec:
replicas: 1
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
containers:
- name: demo
image: yashschandra/demo
imagePullPolicy: Always
ports:
- containerPort: 9999
volumes:
- name: varlog
hostPath:
path: /var/log
My question is -
What am I missing in my golang application because of which fluentd is not receiving logs from app (ie. why don't I see "starting server" in my kibana)?
I have tried both logging to stdout as well as logging to a file but none is working.
I feel it has something to do with pipelining logs correctly to fluentd but almost all articles I read on the topic does not do anything special
references -
https://www.metricfire.com/blog/logging-for-kubernetes-fluentd-and-elasticsearch/
https://medium.com/kubernetes-tutorials/cluster-level-logging-in-kubernetes-with-fluentd-e59aa2b6093a