Rename subkey in json output - fluentd

Question

Is it possible to rename a subkey in JSON or add a new subkey?
For example, I have this log output:
{
"kubernetes": {
"pod_name": "kube-apiserver-tst",
"namespace_name": "kube-system",
"pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
"labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
}
I know that with record_transformer I can add a new key:
<record>
pod_labels "something ..."
</record>
but it seems that it can only create a new key at the root of JSON:
{
"kubernetes": {
"pod_name": "kube-apiserver-tst",
"namespace_name": "kube-system",
"pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
"labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
"pod_labels": "something ..."
}
But can I make it look like this?
{
"kubernetes": {
"pod_name": "kube-apiserver-tst",
"namespace_name": "kube-system",
"pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
"labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
"pod_labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
}
or this:
{
"kubernetes": {
"pod_name": "kube-apiserver-tst",
"namespace_name": "kube-system",
"pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
"pod_labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
}

Answer 1

The JSON is not valid. A closing curly brace is missing.
Here's the valid JSON:
{
"kubernetes": {
"pod_name": "kube-apiserver-tst",
"namespace_name": "kube-system",
"pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
"labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
}
}
Minified JSON (echo '{JSON}' | jq -c .):
{"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","labels":{"component":"kube-apiserver","tier":"control-plane"}}}
The record_transformer filter plugin can be used with Ruby support (via enable_ruby option) to manipulate an existing key and then the unwanted keys can be removed with its remove_keys option.
Here's the sample config:
<filter debug.test>
#type record_transformer
enable_ruby true
<record>
temp ${ l = record["kubernetes"]["labels"]; record["kubernetes"]["pod_labels"] = l; nil; }
</record>
remove_keys temp, $.kubernetes.labels
</filter>
Here's the complete test:
fluent.conf
<source>
#type forward
</source>
<filter debug.test>
#type record_transformer
enable_ruby true
<record>
temp ${ l = record["kubernetes"]["labels"]; record["kubernetes"]["pod_labels"] = l; nil; }
</record>
remove_keys temp, $.kubernetes.labels
</filter>
<match debug.test>
#type stdout
</match>
Start fluentd with this config:
fluentd -c fluent.conf
On another terminal, send an event with fluent-cat (echo '{JSON}' | fluent-cat debug.test):
echo '{"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","labels":{"component":"kube-apiserver","tier":"control-plane"}}}' | fluent-cat debug.test
In fluentd logs, you should see the desired output:
2022-02-16 23:08:25.919967225 +0500 debug.test: {"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","pod_labels":{"component":"kube-apiserver","tier":"control-plane"}}}
Formatted output with jq (echo '{JSON}' | jq .):
echo '{"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","pod_labels":{"component":"kube-apiserver","tier":"control-plane"}}}' | jq .
Output:
{
"kubernetes": {
"pod_name": "kube-apiserver-tst",
"namespace_name": "kube-system",
"pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
"pod_labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
}
}