Best practice for JSON logging in Kubernetes - docker

Question

I'm developing an app whose logs contain custom fields for metric purposes.
Therefore, we produce the logs in JSON format and send them to an Elasticsearch cluster.
We're currently working on migrating the app from a local Docker node to our organization's Kubernetes cluster.
Our cluster uses Fluentd as a DaemonSet, to output logs from all pods to our Elasticsearch cluster.
The setup is similar to this: https://medium.com/kubernetes-tutorials/cluster-level-logging-in-kubernetes-with-fluentd-e59aa2b6093a
I'm trying to figure out what's the best practice to send logs from our app. My two requirements are:
That the logs are formatted correctly in JSON format. I don't want them to be nested in the msg field of the persisted document.
That I can run kubectl logs -f <pod> and view the logs in readable text format.
Currently, if I don't do anything and let the DaemonSet send the logs, it'll fail both requirements.
The best solution I thought about is to ask the administrators of our Kubernetes cluster to replace the Fluentd logging with Fluentbit.
Then I can configure my deployment like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-app
labels:
app: example-app
annotations:
fluentbit.io/parser-example-app: json
fluentbit.io/exclude-send-logs: "true"
spec:
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
containers:
- name: example-app
image: myapp:1.0.0
volumeMounts:
- name: app-logs
mountPath: "/var/log/app"
- name: tail-logs
image: busybox
args: [/bin/sh, -c, 'tail -f /var/log/example-app.log']
volumeMounts:
- name: app-logs
mountPath: "/var/log/app"
volumes:
- name: app-logs
emptyDir: {}
Then the logs are sent to the Elasticsearch in correct JSON format, and I can run kubectl logs -f example-app -c tail-logs to view them in a readable format.
Is this the best practice though? Am I missing a simpler solution?
Is there an alternative supported by Fluentd?
I'll be glad to here your opinion :)

Answer 1

There isn't really a good option here that isn't going to chew massive amounts of CPU. The closest things I can suggest other than the solution you mentioned above is inverting it where the main output stream is unformatted and you run Fluent* (usually Bit) are a sidecar on a secondary file stream. That's no better though.
Really most of us just make the output be in JSON format and on the rare occasions we need to manually poke at logs outside of the normal UI (Kibana, Grafana, whatever), we just deal with the annoyance.
You could also theoretically make your "human" format sufficiently machine parsable to allow for querying. The usual choice there is "logfmt", aka key=value pairs. So my log lines on logfmt-y services look like timestamp=2021-05-15T03:48:05.171973Z level=info event="some message" otherkey=1 foo="bar baz". That's simple enough to read by hand but also can be parsed efficiently.