I am using Raspberry Pi as Thingsboard gateway and pushing sensor readings to server successfully. The Server may not be always online. Where does the data goes in case of server goes offline for long duration? Is there any storage in the Thingsboard gateway?
The "Thingsboard gateway" doesn't have database like Thingsboard Server application but it stores the MQTT messages in case of disconnection from the remote server.
As mentioned the Thingsboard gateway is on Raspberry Pi so go to following directory and make the changes:-
sudo nano /etc/tb-gateway/conf/tb-gateway.yml
there you will find the "gateway.connection.maxInFlight" which represents maximum amount of pending publish messages. Pending messages are messages that are either not sent due to connection problem or not yet confirmed due to high load on Thingsboard Server. Its default value is 1000 (messages). In case of per minute publish cycle ~16 hours of disconnection will be handled by the system.
The persistence settings are configured in the /etc/tb-gateway/conf/tb-gateway.yml file. You can have either “file” or “memory” message persistence. In the latter case data is only memorized in dynamic memory of tb-gateway service, so the following considerations apply only for the "file" option.
The path where data will be stored in case of connection issues between the gateway and the server is also specified in tb-gateway.yml. Default location in Raspbain is: /usr/share/tb-gateway/bin/storage/
Related
We are facing an issue in the customer premises where the gateway which is connected to the meters send data to ThingsBoard using the Node-Red.
To simulate the functionality at the actual client premises created Node-Red nodes which sends data to ThingsBoard. But the data which is received at the ThingsBoard is inconsistent and not at the regular interval which is been set as 30 secs in the Node-Red.
What could be the issue?
Imagine an MQTT broker with remote clients connected, which continuously send QoS 2 data - the standard situation. Clients are configured with "cleansession false" - they have a queue to send messages in case of a connection failure.
On the server, local clients subscribe to topics to receive messages.
Server load:
Launch the MQTT Broker
Running local clients
Connecting remote clients and receiving data from the queue
What if the third point occurs before the second? Are there standard solutions? How not to lose the first messages?
Assuming you are talking about all later reboots of the broker, not the very first time the system is started up then the broker should have stored the persistent subscription state of the clients to disk before it was shutdown and restored this when it restarted. This means that it should queue messages for the local clients.
Also you can always use a firewall to stop the remote client being able to connect until all the local clients have started, this would solve the very first startup issue as well.
We have an IoT based application device which is configured to communication with our Dashboard via MQTT bridge from Various service providers like Google, AWS and Azure.
So the flow is:
Device start TLS session with service provider.
Subscribe to a particular topic and wait for messages from the
service provider with 5 second timeout.
Dashboard publishes messages to same topic periodically.
IoT service provider broadcast it to all devices subscribed.
Publish and subscribe messages are with MQTT QOS 1 services.
Observation:
AWS and Azure works fine with above flow, but device stop receiving messages from Google MQTT bridge after 3-5 successful iterations even though our dashboard is publishing messages to Google IoT MQTT bridge.
For Google, we have identified that control flow is different when compared with Azure and AWS.
For Google, we need to subscribe and un-subscribe for a given topic every-time before waiting to receive message while for AWS and Azure we need to subscribe once during opening a MQTT connection.
Issue:
Sometime 5 sec device timeout occurs as it could not receive messages for subscribed topic from Google MQTT bridge. Adding multiple retries to overcome timeout issue was unsuccessful as issue still persist as device could not receive message from Google MQTT bridge after 45-60sec of device operation after powering on.
Is there is constraint with Google MQTT bridge to receive messages periodically without subscribing it every-time?
How can device receive messages without timing out (5 sec) from Google MQTT bridge?
Is there any workaround to recover a device once it got timed out with establishing MQTT reconnection?
I am using google iot core as well,the device side code for the mqtt client is golang while using paho mqtt package. this client package support OnConnect handler which while using this handler I achieve the recovery which I think you are looking for.
Via this handler I am re-subscribing to the "config" topic.
I think that google does not save the subscriptions which the clients are subscribed to and therefore the client needs to re-subscribe upon successful connection
Here's the golang code I've used (inspired by gingi007's answer, thank you!)
var onConn MQTT.OnConnectHandler
onConn = func(client MQTT.Client) {
fmt.Println("connected")
client.Subscribe(topic.Config, 1, handlerFunc)
}
mqttOpts.SetOnConnectHandler(onConn)
client := MQTT.NewClient(mqttOpts)
this way config updates keep flowing to my device, while if you subscribe outside of the onConnectHandler you'll just receive one config update when you connect.
I have an existing application which was running on solace jar v7.1.2 execute in pub/sub mode. Now we have upgraded to v10.1.1 and as part of implementing DR setup(Disaster Recovery), I have added one more host in the configuration with comma separated.
The application could connect to the primary host successfully, but during the switch-over, (ie from primary to DR) the application had failed to connect and i have received the below error. It connects to DR host if I restart my application.
com.solacesystems.jcsmp.JCSMPErrorResponseException: 400: Unknown Flow Name [Subcode:55]
at com.solacesystems.jcsmp.impl.flow.PubFlowManager.doPubAssuredCtrl(PubFlowManager.java:266)
at com.solacesystems.jcsmp.impl.flow.PubFlowManager.notifyReconnected(PubFlowManager.java:452)
at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2097)
... 5 more
|EAI-000376|||ERROR| |EAI-000376 JMS Exception occurred, Description: `Error sending message - unknown flow name ((JCSMPTransportException)
Need help to understand if we need to have some configuration to do the reconnect to the DR host for a smooth switch over.
In Solace JMS API versions earlier than 7.1.2.226, any sessions on which the clients have published Guaranteed messages will be destroyed after a DR switch‑over. To indicate the disconnect and loss of publisher flow the JMS API will generate this exception. Upon receiving these exceptions, the client application should create a new session. After a new session is established, the client application can republish any Guaranteed messages that had been sent but not acked on the previous session, as these message might not have been persisted and replicated.
However, this behavior was improved in version 7.1.2.226 and later so that the API handles this transparently. It is no longer required to implement code to catch this exception. Can you please verify that the application is not using an API earlier 7.1.2.226? This can be done by enabling debug-level logs.
As Alexandra pointed out, when using guaranteed messaging, as of version 7.1.2 the Solace JMS API guarantees delivery even in the case of failover. It is normal to receive INFO-level log messages that say "Error Response (400) - Unknown Flow Name", this does not indicate a problem, but exceptions (with stack traces) are a problem and indicate that delivery is not guaranteed.
Background: if the connection between the client and the broker (on the Solace server) is terminated unexpectedly, the broker maintains the flow state — but only for three minutes. The state is also copied to the HA mate broker to support failover (but not to the replication mate). If the client reconnects within three minutes, it can resume where it left off. If it reconnects after three minutes, the server will respond with the following (which will be echoed to the logs):
2019-01-04 10:00:59,999 INFO [com.solacesystems.jcsmp.impl.flow.PubFlowManager] (Context_2_Thread_reconnect_service) Error Response (400) - Unknown Flow Name
2019-01-04 10:00:59,999 INFO [com.solacesystems.jcsmp.impl.PubADManager] (Context_2_Thread_reconnect_service) Unknown Publisher Flow (flowId=36) recovered: 1 messages renumbered and resent (lastMessageIdSent =0)
That's okay: the client JMS library will automatically resend whatever messages are necessary, so guaranteed messaging is still guaranteed.
Also, just to confirm, the jar name indicates the version, so sol-jms-10.1.1.jar uses version 10.1.1.
I am using Google IOT core with mongoose os. I wanted to update device connection status to firestore. But i am unable to find event which reports mqtt connection status to pub/sub like when device disconnects or reconnect i.e if device is offline or not.
I am stuck on this problem for days.Any help will be appreciated
Update
As #devunwired mentioned in this response it is now possible to monitor Stackdriver logs for disconnect events. You must have at a minimum enabled INFO level logging on your project in IoT Core > Registries > [your registry] > Edit Registry > Select "Info" log level > Click save.
Original Response
There are a few values you can look at that are tracked in device configuration metadata that you could use to know when a device last was online:
Last Configuration Send time - sent anytime your device connects /
configuration is posted
Last Event Time - Last time an event was sent from the device
Last State Time - Last time state was sent from the device
Last Heartbeat time - Last time MQTT heartbeat was sent
To get you started, here is an example using API explorer that you can fill-in with your project ID, region, registry, and device to query for a specific device's metadata.
For 1...3 you have control over these through device manager and by publishing data. MQTT heartbeat is updated if your device sends an MQTT_PINGREQ message during the "ping period" without other messages getting sent.
At any rate, you could use any of these update time values to see the last time a device was online / functioning. You could query the states of your devices after listing the devices in a registry and could update a Firebase RTDB periodically if that's how you want to report (e.g. using AppEngine TaskQueue). Note that you also just can get these "last connected" values from the Google Cloud Console.
It was said before but we don't have an event for disconnect, just configuration ack, which generally is the connection event. If you want to share state between a device and the device manager, use state messages.
Unfortunately, there's no built in way to do this right now as there aren't events on this state.
However, you could implement a hack by sending a message on connect/disconnect from the device that you have a Cloud Function subscribed to the Pub/Sub topic listening for. It's not perfect as it would fail in the case where the device disconnected unexpectedly.
There currently is no way to do this, that i've been able to find (a year later after this original post). I posted a question here on SO regarding this as well, with more details and link to example code I had to use for handling this:
Google Core IoT Device Offline Event or Connection Status
The AWS IoT platform publishes messages on a special MQTT topic (prefixed with $aws) when your device connects/disconnects. You can easily use these to monitor these events - however, you should be aware that the MQTT protocol is designed to be robust to a poor networking conditions and the broker on the AWS side probably doesn't think it's a bit deal to disconnect a client. The broker expects that the client will just reconnect and queue messages for a moment during that process (which can be a big deal on a microcontroller).
All that being said, the AWS topics you would watch are:
$aws/events/presence/connected/{clientId}
and
$aws/events/presence/disconnected/{clientId}
and the documentation for these (and other) lifecycle events are located: https://docs.aws.amazon.com/iot/latest/developerguide/life-cycle-events.html