How can I define or program an app's error channel that will receive all messages that have failed in processors/sinks?
The documentation says the following:
When the number of retry attempts has exceeded the maxAttempts value, the exception and the failed message will become the payload of a message and be sent to the application’s error channel. By default, the default message handler for this error channel logs the message. You can change the default behavior in your application by creating your own message handler that subscribes to the error channel.
After that documentation talks about enabling dead letter queues in binders. If I understand that correctly, the whole concept means that I can write my own handler that will subscribe to the DLQ of the binder and receive the messages.
I am curious, if it is possible to define a separate stream that will receive failed messages, or write an additional app that will receive those failed payloads and process them how it wants without utilizing the DLQ of the binder?
Assuming you've enabled DLQ and depending on the binder implementation in use, you may have to create a separate application to drain and process messages from DLQ.
The recommended approaches for rabbit-binder, for example, can be found in the docs.
Related
Here is our scenario:
We are publishing messages to a topic, and have a queue subscribed to that topic. We have several node.js consumers connected to that queue to process incoming messages, via the solclient package.
What we want to do is process the incoming messages, and after successfully processing the message, acknowledge the message so that it is removed from the queue. The challenge we're having is how to deal with error conditions. We are trying to figure out how to flag to the broker that the message failed to be processed? The expectation would be that the broker would then attempt to send to a consumer again, and once max redeliveries is hit, it moves to the DMQ.
I don't believe there's a way in Solace for a consumer to "NACK" a received message to signal an error in processing. I believe your option would be to unbind from the queue (i.e. disconnect() the Flow object, or MessageConsumer in the Node.js API) which will place allow any unacknowledged messages back on the queue and available for redelivery, and then rebind to the queue.
Alternatively, if you do your processing in the message received callback method, you could throw an exception in there, and that should (?) accomplish the same thing. But probably a lot less graceful.
https://docs.solace.com/Solace-PubSub-Messaging-APIs/Developer-Guide/Creating-Flows.htm
https://docs.solace.com/API-Developer-Online-Ref-Documentation/js/solace.MessageConsumer.html#disconnect
How to know the remote device is online using mqtt (as a device simulator I am using MQTTfx). After publishing to a topic I am not getting any acknowledgement message and in the callbacks I am getting the same payload data that is published.
You can not find out is another MQTT client is online at a protocol level, in fact at a Pub/Sub pattern level a publishing client should not care is 0 or an infinite number of other clients are subscribed to the topic that message is published on.
The callbacks are only an indication that the message has been successfully delivered to the broker not to any of the subscribers. There is no end to end delivery notification (because there may be no subscribers to a given topic).
If you need to know the status of devices then you need to implement this yourself, the usual pattern is to use a status topic with a retained message and the LWT to clear this if the client crashes and setting it manually when it starts up or shuts down cleanly.
I have an existing application which was running on solace jar v7.1.2 execute in pub/sub mode. Now we have upgraded to v10.1.1 and as part of implementing DR setup(Disaster Recovery), I have added one more host in the configuration with comma separated.
The application could connect to the primary host successfully, but during the switch-over, (ie from primary to DR) the application had failed to connect and i have received the below error. It connects to DR host if I restart my application.
com.solacesystems.jcsmp.JCSMPErrorResponseException: 400: Unknown Flow Name [Subcode:55]
at com.solacesystems.jcsmp.impl.flow.PubFlowManager.doPubAssuredCtrl(PubFlowManager.java:266)
at com.solacesystems.jcsmp.impl.flow.PubFlowManager.notifyReconnected(PubFlowManager.java:452)
at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2097)
... 5 more
|EAI-000376|||ERROR| |EAI-000376 JMS Exception occurred, Description: `Error sending message - unknown flow name ((JCSMPTransportException)
Need help to understand if we need to have some configuration to do the reconnect to the DR host for a smooth switch over.
In Solace JMS API versions earlier than 7.1.2.226, any sessions on which the clients have published Guaranteed messages will be destroyed after a DR switch‑over. To indicate the disconnect and loss of publisher flow the JMS API will generate this exception. Upon receiving these exceptions, the client application should create a new session. After a new session is established, the client application can republish any Guaranteed messages that had been sent but not acked on the previous session, as these message might not have been persisted and replicated.
However, this behavior was improved in version 7.1.2.226 and later so that the API handles this transparently. It is no longer required to implement code to catch this exception. Can you please verify that the application is not using an API earlier 7.1.2.226? This can be done by enabling debug-level logs.
As Alexandra pointed out, when using guaranteed messaging, as of version 7.1.2 the Solace JMS API guarantees delivery even in the case of failover. It is normal to receive INFO-level log messages that say "Error Response (400) - Unknown Flow Name", this does not indicate a problem, but exceptions (with stack traces) are a problem and indicate that delivery is not guaranteed.
Background: if the connection between the client and the broker (on the Solace server) is terminated unexpectedly, the broker maintains the flow state — but only for three minutes. The state is also copied to the HA mate broker to support failover (but not to the replication mate). If the client reconnects within three minutes, it can resume where it left off. If it reconnects after three minutes, the server will respond with the following (which will be echoed to the logs):
2019-01-04 10:00:59,999 INFO [com.solacesystems.jcsmp.impl.flow.PubFlowManager] (Context_2_Thread_reconnect_service) Error Response (400) - Unknown Flow Name
2019-01-04 10:00:59,999 INFO [com.solacesystems.jcsmp.impl.PubADManager] (Context_2_Thread_reconnect_service) Unknown Publisher Flow (flowId=36) recovered: 1 messages renumbered and resent (lastMessageIdSent =0)
That's okay: the client JMS library will automatically resend whatever messages are necessary, so guaranteed messaging is still guaranteed.
Also, just to confirm, the jar name indicates the version, so sol-jms-10.1.1.jar uses version 10.1.1.
I am using MQTT with qos 1 which means messages may be sent to client more than once? How can i prevent client from handle same messages twice?
The only way to know for sure if you have received the message twice is to include an ID in the message payload and maintain a list of processed messages in the client.
The PAHO API includes a isDuplicate() (Javascript version) method on received messages, but this is only an indication that the message may be a duplicate, because the original message may never have arrived.
The company I am working for has evaluated MQTT and decided to use it as a core messaging platform for a large scale system. The main reason is how compact the protocol is and how easy it can actually be implemented. I have a single issue with MQTT though and I'm seeking for an answer to the following question:
QoS1 and QoS2 messages require confirmation from the client. The only thing I know about the message (identifying it) when receiving PUBACK, PUBREC, PUBREL and PUBCOMP is messageId and the clientId. Message id is an unsigned int16 so the max value is 65535. It doesn't seem to be large enough for long running clients, say a year, sending 15 QoS2 messages an hour.
I am not quite sure if there's any other way to identify the message? I would like to be as compliant with the standard as possible.
Probably the first point to make clear is that message IDs are handled on a per client and per direction basis. That is to say that the broker will create a message ID for each outgoing message with QoS>0 for each client that is connected and these message IDs will be completely independent of any other message IDs used for the same message published to other clients. Likewise, each client generates its own message IDs for messages that it sends.
The message ID doesn't have to be unique, so your client sending 15 messages per hour with QoS level 2 would simply overflow at some point. The real limitation is that there can only be a maximum of 65535 messages per direction "in flight" at once (i.e. part way through the message handshake). Once a message with a given ID has been fully processed then that message ID can be reused.
Another way of looking at it is to consider how it would work if your client only ever had one message in flight at once, whether because of the rate the messages are being transmitted or by design in the way you handle the messages. In this case, you could keep message ID set to 1 for every single message because there is never a chance that there will be a duplicate.
If you wish to support having multiple messages in flight at once it would be relatively straightforward to check there are no message ID duplicates before you assign a new one.
Because the message ID is per client, if you send a single message to >65535 clients there will be no chance of message ID collisions. If you send >65535 messages to each client at once and the message flows aren't complete then there will be problems.
Answering the comment "I have noticed that every MQTT broker tends to deliver only the last QoS1/2 message":
The broker will only send messages to clients it knows about. If you connect for the first time there is no way to get messages from the past, with one exception: retained messages. If a message is set to retained then it is a "last known good" value. When a new client subscribes it will be sent the retained message immediately which makes it useful for things that are updated infrequently. I suspect this is what you are referring to. If you want a client to have messages queued when it is not connected then you must connect with the "clean session" option disabled to make the client persistent. You must also use QoS>0 subscriptions and QoS>0 publications. When your client reconnects (with clean session still set to disabled), the queued messages will be delivered. You can normally configure the number of messages to queue in this way in the broker, where any further messages will be discarded. An important point is that queueing messages for a client that has not previously connected is not supported by design.
For delivering more messages at QOS1 or QOS2, you should use concept of persistant memory. In this when ever a subscriber is not available the message get stored in persistant memory and deliver once subscriber is connected. You can do this at QOS0 also after configuring mosquitto.conf file.