Influxdb Json (MQTT) rename field with wildcards - mqtt

Question

I need help with an issue I am having sending the data of my Zigbee thermometers to influx, via telegraf.
this is the path:
Zigbee Sonoff SNZB-02 --> Tasmota ZBBridge --> MQTT --> Telegraf --> InfluxDB
The id of the zigbee thermometer: 0x4EF9 might change since it's randomly assigned to the device, in Tasmota I am able to assign a "friendly name", in this case: ZB_Sonoff_Temp01
With simple tasmota devices I have no issues, I have the single entry of the device in the MQTT Topic and Telegraf plays nicely with those.
My issues is with the data from the Zigbee Bridge, since it has a single topic and the output in Influx is a bit difficult to work with:
Example MQTT message for Zigbee thermometer:
tele/tasmota_ABDCEF/ZB_Sonoff_Temp01/SENSOR {"ZbReceived":{"0x4EF9":{"Device":"0x4EF9","Name":"ZB_Sonoff_Temp01","Humidity":94.84,"Endpoint":1,"LinkQuality":34}}}
the data is in Json format as you can see,
in Telegraf I am using mqtt_consumer, here is the config:
/etc/telegraf/telegraf.d/mqtt.conf
[[inputs.mqtt_consumer]]
servers = ["tcp://192.168.10.10:1883"]
## Topics that will be subscribed to.
topics = [
"tele/tasmota_ABCDEF/ZB_Sonoff_Temp01/SENSOR"
]
qos = 0
connection_timeout = "30s"
username = "user"
password = "password"
data_format = "json"
[[outputs.influxdb]]
urls = ["http://localhost:8086"]
database = "test_temp"
# skip_database_creation = true
and this is my /etc/telegraf/telegraf.conf:
[global_tags]
[agent]
logfile = "/var/log/telegraf/telegraf.log"
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
this is the data in influx:
time ZbReceived_0x4EF9_Endpoint ZbReceived_0x4EF9_Humidity ZbReceived_0x4EF9_LinkQuality ZbReceived_0x4EF9_Temperature host topic
---- -------------------------- -------------------------- ----------------------------- ----------------------------- ---- -----
2021-12-24T15:43:55.26962955Z 1 99.99 26 32.24 influxdb-test tele/tasmota_ABCDEF/ZB_Sonoff_Temp01/SENSOR
2021-12-24T15:43:55.560162845Z 1 21 25.18 influxdb-test tele/tasmota_ABDCEF/ZB_Sonoff_Temp01/SENSOR
which could be ok, since I am able to choose the device via the "topic" field, but the problems is that the field "ZbReceived_0x4EF9_Temperature" is not "sane" in case the devices change ID when re-associating with zigbee, which might happen..
the workaround I found is to add a rename for the fields:
[[processors.rename]]
[[processors.rename.replace]]
field = "ZbReceived_0x4EF9_Temperature"
dest = "Temperature"
[[processors.rename.replace]]
field = "ZbReceived_0x4EF9_Humidity"
dest = "Humidity"
[[processors.rename.replace]]
field = "ZbReceived_0x4EF9_Endpoint"
dest = "Endpoint"
[[processors.rename.replace]]
field = "ZbReceived_0x4EF9_LinkQuality"
dest = "LinkQuality"
which changes the fields as I want (there is no Humidity but it's not always pushed, so it's ok, I am dropping the db between changes):
time Endpoint LinkQuality Temperature host topic
---- -------- ----------- ----------- ---- -----
2021-12-24T15:47:09.992947108Z 1 21 23 influxdb-test tele/tasmota_ABCDEF/ZB_Sonoff_Temp01/SENSOR
2021-12-24T15:47:25.868416967Z 1 13 27.06 influxdb-test tele/tasmota_ABCDEF/ZB_Sonoff_Temp01/SENSOR
I don't like this solution very much since it has the device ids hardcoded in telegraf config, so when I add or change a sensor I need to edit telegraf.
the problem I now have is that I would like to find a wildcard or a method to change the fields independently from the device id, like:
[[processors.rename]]
[[processors.rename.replace]]
field = "*_Temperature"
dest = "Temperature"
but I am not able to find it, I've read all the docs of the methods (also strings) but I could not find a way to achieve that..
do you have any tip that could help me?
thank you very much and happy holidays!

Answer 1

I needed to use processors.regex.field_rename, but I had an older version of telegraf than the latest ((1.21.1-1) over (1.20.2-1)), and so field_rename was not available.
The regex I've used:
[[processors.regex]]
[[processors.regex.field_rename]]
pattern = '(^ZbReceived_)\w+_'
replacement = "${2}"
the result:
time Endpoint Humidity LinkQuality Temperature host topic
---- -------- -------- ----------- ----------- ---- -----
2021-12-24T23:51:48.289677669Z 1 0 25.26 influxdb-test tele/tasmota_ABCDEF/ZB_Sonoff_Temp01/SENSOR