Azure IoT Edge Module Inter-module message repeat same message - azure-iot-edge

Question

I have 2 Auzre IoT Edge Module running on Linux (Ubuntu 20.04) device. Module A (inzightedge) send message to Module B (videoedge) via edgeHub with the following routes.
"routes": {
"InzightEdgeToIoTHub": "FROM /messages/modules/inzightedge/outputs/* INTO $upstream",
"inzight2video": "FROM /messages/modules/inzightedge/outputs/videoOutput INTO BrokeredEndpoint("/modules/videoedge/inputs/videoInput")"
},
In the videoedge (Module B) module, Program.cs:
static async Task Init()
{
MqttTransportSettings mqttSetting = new MqttTransportSettings(TransportType.Mqtt_Tcp_Only);
ITransportSettings[] settings = { mqttSetting };
_ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
await _ioTHubModuleClient.OpenAsync();
ConsoleLog.LogInformation("Init", "IoT Hub module client initialized.");
await _ioTHubModuleClient.SetInputMessageHandlerAsync("videoInput", VideoInputMessage, _ioTHubModuleClient);
ConsoleLog.LogInformation("Init", "Start initialization");
}
static async Task<MessageResponse> VideoInputMessage(Message message, object userContext)
{
var moduleClient = userContext as ModuleClient;
int counterValue = Interlocked.Increment(ref _videoRequestCounter);
if (moduleClient == null)
{
throw new InvalidOperationException("UserContext doesn't contain " + "expected values");
}
byte[] messageBytes = message.GetBytes();
string messageString = Encoding.UTF8.GetString(messageBytes);
ConsoleLog.LogInformation("VideoInputMessage", $"Received message: {counterValue}, Body: {messageString}");
if (!string.IsNullOrEmpty(messageString))
{
// Do something to handle the message
}
return MessageResponse.Completed;
}
When Module A (inzightedge) send a message to Module B (videoedge), i.e. "videoInput".
Module B can get the first message, but it may not get the second one. However, Module B keep receiving the first message.
The log in edgeHub shows that the message is return to the message queue again. Here is the log:
<6> 2022-06-16 05:03:29.513 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-06-16 05:08:29.512 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-06-16 05:08:29.513 +00:00 [INF] - Error authenticating token for SE50-02/inzightedge because the token is expired or could not be parsed
<6> 2022-06-16 05:08:29.514 +00:00 [INF] - Credentials for client SE50-02/inzightedge are not valid.
<4> 2022-06-16 05:08:29.514 +00:00 [WRN] - Reauthenticating client SE50-02/inzightedge failed, removing client connection
<6> 2022-06-16 05:08:29.514 +00:00 [INF] - Unable to re-authenticate SE50-02/inzightedge, dropping client connection.
<6> 2022-06-16 05:08:29.514 +00:00 [INF] - Removing device connection for device SE50-02/inzightedge with removeCloudConnection flag 'True'.
<4> 2022-06-16 05:08:29.516 +00:00 [WRN] - Closing connection for device: SE50-02/inzightedge, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device SE50-02/inzightedge.,
<6> 2022-06-16 05:08:29.516 +00:00 [INF] - Disposing MessagingServiceClient for device Id SE50-02/inzightedge because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device SE50-02/inzightedge.
<6> 2022-06-16 05:08:29.517 +00:00 [INF] - Closing device proxy for device Id SE50-02/inzightedge
<6> 2022-06-16 05:08:29.519 +00:00 [INF] - Closing receiver in cloud proxy 6500cd9d-b22e-47a7-a9de-f0a998b963d0 for SE50-02/inzightedge
<6> 2022-06-16 05:08:29.519 +00:00 [INF] - Closed cloud proxy 6500cd9d-b22e-47a7-a9de-f0a998b963d0 for SE50-02/inzightedge
<6> 2022-06-16 05:08:29.519 +00:00 [INF] - Device connection removed for device SE50-02/inzightedge
<6> 2022-06-16 05:08:30.080 +00:00 [INF] - Updated reported properties for SE50-02/$edgeHub
<6> 2022-06-16 05:08:30.083 +00:00 [INF] - Client SE50-02/inzightedge in device scope authenticated locally.
<6> 2022-06-16 05:08:30.084 +00:00 [INF] - Successfully generated identity for clientId SE50-02/inzightedge and username ubuntu/SE50-02/inzightedge/?api-version=2020-09-30&DeviceClientType=.NET%2F1.41.0%20%28.NET%20Core%203.1.25%3B%20Linux%205.4.0-117-generic%20%23132-Ubuntu%20SMP%20Thu%20Jun%202%2000%3A39%3A06%20UTC%202022%3B%20X64%29
<6> 2022-06-16 05:08:30.084 +00:00 [INF] - ClientAuthenticated, SE50-02/inzightedge, 27d3c149
<6> 2022-06-16 05:08:30.084 +00:00 [INF] - New device connection for device SE50-02/inzightedge
<6> 2022-06-16 05:08:30.084 +00:00 [INF] - Client SE50-02/inzightedge connected to edgeHub, processing existing subscriptions.
<6> 2022-06-16 05:08:30.085 +00:00 [INF] - Bind device proxy for device SE50-02/inzightedge
<6> 2022-06-16 05:08:30.085 +00:00 [INF] - Binding message channel for device Id SE50-02/inzightedge
<6> 2022-06-16 05:08:30.085 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.086 +00:00 [INF] - Attempting to connect to IoT Hub for client SE50-02/inzightedge via AMQP...
<6> 2022-06-16 05:08:30.088 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates, ModuleMessages for client SE50-02/inzightedge.
<6> 2022-06-16 05:08:30.088 +00:00 [INF] - Set subscriptions from session state for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.091 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates, ModuleMessages for client SE50-02/inzightedge.
<6> 2022-06-16 05:08:30.091 +00:00 [INF] - Set subscriptions from session state for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.830 +00:00 [INF] - Cloud connection for SE50-02/inzightedge is True
<6> 2022-06-16 05:08:30.830 +00:00 [INF] - Connection status for SE50-02/inzightedge changed to ConnectionEstablished
<6> 2022-06-16 05:08:30.830 +00:00 [INF] - Client SE50-02/inzightedge connected to cloud, processing existing subscriptions.
<6> 2022-06-16 05:08:30.830 +00:00 [INF] - Skipping SE50-02/inzightedge for subscription processing, as it is currently being processed.
<6> 2022-06-16 05:08:30.830 +00:00 [INF] - Created cloud proxy for client SE50-02/inzightedge via AMQP, with client operation timeout 20 seconds.
<6> 2022-06-16 05:08:30.831 +00:00 [INF] - Initialized cloud proxy 2adf6f8a-884e-4968-a735-a23edabed330 for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.831 +00:00 [INF] - Created cloud connection for client SE50-02/inzightedge
<6> 2022-06-16 05:08:30.847 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.847 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.847 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:08:30.847 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:08:35.616 +00:00 [INF] - Updated reported properties for SE50-02/$edgeHub
<6> 2022-06-16 05:10:47.999 +00:00 [INF] - New token received on the Cbs link
<6> 2022-06-16 05:10:48.000 +00:00 [INF] - Token updated for SE50-02/SimulatedTemperatureSensor
<6> 2022-06-16 05:10:48.160 +00:00 [INF] - New token received on the Cbs link
<6> 2022-06-16 05:10:48.160 +00:00 [INF] - Token updated for SE50-02/SimulatedTemperatureSensor
<6> 2022-06-16 05:13:29.512 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2022-06-16 05:15:59.303 +00:00 [WRN] - Closing connection for device: SE50-02/inzightedge, Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.,
<6> 2022-06-16 05:15:59.303 +00:00 [INF] - Disposing MessagingServiceClient for device Id SE50-02/inzightedge because of exception - Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.
<6> 2022-06-16 05:15:59.303 +00:00 [INF] - Setting device proxy inactive for device Id SE50-02/inzightedge
<6> 2022-06-16 05:15:59.303 +00:00 [INF] - Removing device connection for device SE50-02/inzightedge with removeCloudConnection flag 'True'.
<6> 2022-06-16 05:15:59.304 +00:00 [INF] - Closing receiver in cloud proxy 2adf6f8a-884e-4968-a735-a23edabed330 for SE50-02/inzightedge
<6> 2022-06-16 05:15:59.304 +00:00 [INF] - Closed cloud proxy 2adf6f8a-884e-4968-a735-a23edabed330 for SE50-02/inzightedge
<6> 2022-06-16 05:15:59.304 +00:00 [INF] - Device connection removed for device SE50-02/inzightedge
<6> 2022-06-16 05:15:59.305 +00:00 [INF] - Remove device connection for device SE50-02/inzightedge
<6> 2022-06-16 05:15:59.527 +00:00 [INF] - Updated reported properties for SE50-02/$edgeHub
<6> 2022-06-16 05:16:00.207 +00:00 [INF] - Client SE50-02/inzightedge in device scope authenticated locally.
<6> 2022-06-16 05:16:00.207 +00:00 [INF] - Successfully generated identity for clientId SE50-02/inzightedge and username ubuntu/SE50-02/inzightedge/?api-version=2020-09-30&DeviceClientType=.NET%2F1.41.0%20%28.NET%20Core%203.1.25%3B%20Linux%205.4.0-117-generic%20%23132-Ubuntu%20SMP%20Thu%20Jun%202%2000%3A39%3A06%20UTC%202022%3B%20X64%29
<6> 2022-06-16 05:16:00.207 +00:00 [INF] - ClientAuthenticated, SE50-02/inzightedge, 68eb07da
<6> 2022-06-16 05:16:00.208 +00:00 [INF] - New device connection for device SE50-02/inzightedge
<6> 2022-06-16 05:16:00.208 +00:00 [INF] - Client SE50-02/inzightedge connected to edgeHub, processing existing subscriptions.
<6> 2022-06-16 05:16:00.209 +00:00 [INF] - Bind device proxy for device SE50-02/inzightedge
<6> 2022-06-16 05:16:00.209 +00:00 [INF] - Binding message channel for device Id SE50-02/inzightedge
<6> 2022-06-16 05:16:00.209 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.209 +00:00 [INF] - Attempting to connect to IoT Hub for client SE50-02/inzightedge via AMQP...
<6> 2022-06-16 05:16:00.223 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates, ModuleMessages for client SE50-02/inzightedge.
<6> 2022-06-16 05:16:00.224 +00:00 [INF] - Set subscriptions from session state for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Cloud connection for SE50-02/inzightedge is True
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Connection status for SE50-02/inzightedge changed to ConnectionEstablished
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Client SE50-02/inzightedge connected to cloud, processing existing subscriptions.
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Skipping SE50-02/inzightedge for subscription processing, as it is currently being processed.
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Created cloud proxy for client SE50-02/inzightedge via AMQP, with client operation timeout 20 seconds.
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Initialized cloud proxy 35825535-8969-4124-b28b-78422ac6770f for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.255 +00:00 [INF] - Created cloud connection for client SE50-02/inzightedge
<6> 2022-06-16 05:16:00.271 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.271 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.271 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.543 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates, ModuleMessages for client SE50-02/inzightedge.
<6> 2022-06-16 05:16:00.543 +00:00 [INF] - Processing pending subscriptions for SE50-02/inzightedge
<6> 2022-06-16 05:16:00.543 +00:00 [INF] - Set subscriptions from session state for SE50-02/inzightedge
<6> 2022-06-16 05:16:04.749 +00:00 [INF] - Updated reported properties for SE50-02/$edgeHub
<6> 2022-06-16 05:16:05.057 +00:00 [INF] - Updated reported properties for SE50-02/inzightedge
<4> 2022-06-16 05:16:35.617 +00:00 [WRN] - Did not receive ack for message 467c44e4-d5fd-4bc5-a557-bd75c540427a from device/module SE50-02/videoedge
<4> 2022-06-16 05:16:35.618 +00:00 [WRN] - Error sending messages to module SE50-02/videoedge
System.TimeoutException: Message completion response not received
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 499
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.ProcessAsync(ICollection`1 routingMessages, IDeviceProxy dp, CancellationToken token) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 167
<4> 2022-06-16 05:17:06.636 +00:00 [WRN] - Did not receive ack for message 80ee9b81-8ba5-4767-8b80-2a9cc4bb58bd from device/module SE50-02/videoedge
<4> 2022-06-16 05:17:06.637 +00:00 [WRN] - Error sending messages to module SE50-02/videoedge
System.TimeoutException: Message completion response not received
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 499
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.ProcessAsync(ICollection`1 routingMessages, IDeviceProxy dp, CancellationToken token) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 167
<4> 2022-06-16 05:17:08.799 +00:00 [WRN] - Received unknown feedback message from SE50-02/videoedge with lock token 467c44e4-d5fd-4bc5-a557-bd75c540427a and status Complete. Abandoning message.
Retrieved 90 line(s)
When I create a blank new module to receive the message and it has no problem. May someone the potential problem of the videoedge module?

Answer 1

These are the steps to reproduce:
Create a custom module to get output in the form of message using ModuleClient
Create trid for each message and using the for loop to increment it for each message
Add the message body size of 64KB of size
Create blog storage
Attach the data dump to the blob storage
Check the blob storage whether the messages are being received with trid header or not.