Telegraf 1.16 + inputs.modbus plugin: timeout problem - influxdb

Question

I am reading with Telegraf 1.16 some Janitza devices through the inputs.modbus plugin.
Telegraf is started manually and not as a service to ease tests and debugging.
This is the configuration:
Unit1 is a UMG604 that acts as a Gateway: it receives Modbus/TCP messages, and if they don't match its modbus address number, relays them to the following units. These are linked through a RS485 line. That means the communication is half-duplex and the line is quite busy because we are trying to read 350+ registers at any tick (25 registers per device).
These units are read without any problem using two loggers I wrote, one in C, the other in Python/pymodbus, so I can exclude any hardware issue.
Settings are straightforward, and here is a skeleton of Telegraf configuration file:
[agent]
interval="5s" # sample time
round_interval=true # sample at rounded intervals :00, :05, :10, etc
metric_batch_size=1000
metric_buffer_limit=10000
[[inputs.modbus]]
name = "UMG604_Gateway_unit1"
slave_id = 1
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit1", name="Strom-1", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit2"
slave_id = 2
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit2", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit3"
slave_id = 3
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit3", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit4"
slave_id = 4
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit4", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit5"
slave_id = 5
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit5", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit6"
slave_id = 6
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit6", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit7"
slave_id = 7
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit7", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[inputs.modbus]]
name = "UMG103_unit8"
slave_id = 8
timeout = "2s"
busy_retries = 10
busy_retries_wait = "200ms"
controller = "tcp://192.168.2.100:502"
holding_registers = [
{ measurement="Unit8", name="", byte_order="ABCD", data_type="FLOAT32-IEEE", scale=1.0, address=[1325,1326]},
# <another 24 non consecutive registers here>
]
[[outputs.influxdb_v2]]
urls = ["http://localhost:8086"]
token = "XXXXXXX"
organization = "demo_org"
bucket = "demo_bucket"
The problem
The first units in the config file are read quite regularly, but unit 5..8 manifest almost always a timeout:
dial tcp 192.168.2.100:502: i/o timeout
read tcp 192.168.2.XX:XXXX->192.168.2.100:502: i/o timeout
There are not so many parameters to tweak (timeout, busy_retry and busy_retry_wait has been increased), so I don't know if what I experience is a wrong setting or a problem in the modbus plugin.
Question 1. How is Telegraf reading the devices? Are the requests multithreaded, so they are ideally read at the same time?
If this is the case, the culprit could be the gateway UMG604 unit 1: it only accepts 4 connections.
Question 2: Is there a possibility to delay the reading of some of the units 2..8? If it is, I can read the first ones, then after a delay the second block, and then a third. The lack of simultainety is not an issue in my system.
As a workaround I wrote a minimal input.exec that reads and print out a JSON that is fed to Telegraf, but if possible I would like to use a standard solution based only on the standard plugin.
EDIT 1 ############################################
Alas the devices are not accessible for experimentation: I can only ssh a remote server where Telegraf is installed.
I know from experience that there are no particular issues when reading them in different times, so after some comments I assume the problem is due to UMG604 accepting only 4 incoming connection at the same time.
Reading up to 3 devices (in any combination) never generates timeouts.
Any idea about how to activate a delayed reading of some of them? It could be a test that proves the point.
EDIT 2 ############################################
A quick recap: the error IS NOT generated by UMG604 limited number of connections.
All 8 connections open correctly, 4 soon, 4 in a later time.
The problem is the Modbus/RTU reading of UMG103 units: it looks like after the first successful readings they just stop returning data. As I told, tests cannot be done in loco.
There is an error but no return code for the error from getFields (ok=false), and this means Gather function in modbus.go plugin just exits without even retrying to read again:
if err != nil {
mberr, ok := err.(*mb.ModbusError)
if ok && mberr.ExceptionCode == mb.ExceptionCodeServerDeviceBusy && retry < m.Retries {
...
time.Sleep(m.RetriesWaitTime.Duration)
continue # try again, we are inside a for loop
}
# ok is false, so we jump here!
disconnect(m)
m.isConnected = false
return err
}
Alas forcing to retry on ANY error doesn't work.
After an error I also tried closing and reopeningthe connection before retrying, but it doesn't work.