Here is our scenario:
We are publishing messages to a topic, and have a queue subscribed to that topic. We have several node.js consumers connected to that queue to process incoming messages, via the solclient package.
What we want to do is process the incoming messages, and after successfully processing the message, acknowledge the message so that it is removed from the queue. The challenge we're having is how to deal with error conditions. We are trying to figure out how to flag to the broker that the message failed to be processed? The expectation would be that the broker would then attempt to send to a consumer again, and once max redeliveries is hit, it moves to the DMQ.
I don't believe there's a way in Solace for a consumer to "NACK" a received message to signal an error in processing. I believe your option would be to unbind from the queue (i.e. disconnect() the Flow object, or MessageConsumer in the Node.js API) which will place allow any unacknowledged messages back on the queue and available for redelivery, and then rebind to the queue.
Alternatively, if you do your processing in the message received callback method, you could throw an exception in there, and that should (?) accomplish the same thing. But probably a lot less graceful.
https://docs.solace.com/Solace-PubSub-Messaging-APIs/Developer-Guide/Creating-Flows.htm
https://docs.solace.com/API-Developer-Online-Ref-Documentation/js/solace.MessageConsumer.html#disconnect
Related
I have configured a MQTT subscriber in spring using spring mqtt integration. In the handleMessage method I am doing certain business logic which takes time. While testing I noticed that when I am sending bulk number of messages the Broker republishes the same message as an original message (I checked whether the Message payload is duplicate , it was sending as original). The MQTT Broker is publishing the message again even before the Subscriber can send PUCOMP. QOS level is set to 2
You should not be doing long running tasks in the handleMessage callback as this is run on the MQTT Clients network thread.
If you have a long running task you should be handing it off to a separate thread pool to run.
I think I described almost everything I need in title. So there is some WMB flows. And one wait for the answer in queue. I need to throw exception if there will be no message in queue after timeout.
Thank you for your time
Yes it is possibe, but you will need to develop it in your flows. MQ is made for asyncronous communication, so a timeout is not something which is native to it. I can think of 2 possible solutions now:
Use the TimeoutControl and TimeoutNotification nodes in your flows
In the flow which sends the request, after sending the request you add a TimeoutControl node and set up the desired timeout.
Create a new flow, which starts with the TimeoutNotification flow. In that flow you send your timeout error if the response has not yet been received.
And to know which response has been received you can use different methods, for example the flows sending the request and receiving the response could maintain a database table, or you could store this information in a queue as well.
Start waiting for the response after sending the request
Set up the response handler flow to start with an MQ Input followed by an MQ Get node. You listen for the response with the MQ Get, on which you can set a wait interval, that will be your timeout threshold. The MQ Input gets technical messages sent by the request sender flow after sending the request.
This is a worse solution then the first, as you will block a message flow thread while listening for the response.
Or you can just make 1 flow to send the request and receive the response, receiving the response with an MQ Get node.
This is even worse as you will need to turn off transactionality for the MQ Output sending the request.
How can I define or program an app's error channel that will receive all messages that have failed in processors/sinks?
The documentation says the following:
When the number of retry attempts has exceeded the maxAttempts value, the exception and the failed message will become the payload of a message and be sent to the application’s error channel. By default, the default message handler for this error channel logs the message. You can change the default behavior in your application by creating your own message handler that subscribes to the error channel.
After that documentation talks about enabling dead letter queues in binders. If I understand that correctly, the whole concept means that I can write my own handler that will subscribe to the DLQ of the binder and receive the messages.
I am curious, if it is possible to define a separate stream that will receive failed messages, or write an additional app that will receive those failed payloads and process them how it wants without utilizing the DLQ of the binder?
Assuming you've enabled DLQ and depending on the binder implementation in use, you may have to create a separate application to drain and process messages from DLQ.
The recommended approaches for rabbit-binder, for example, can be found in the docs.
I am using MQTT with qos 1 which means messages may be sent to client more than once? How can i prevent client from handle same messages twice?
The only way to know for sure if you have received the message twice is to include an ID in the message payload and maintain a list of processed messages in the client.
The PAHO API includes a isDuplicate() (Javascript version) method on received messages, but this is only an indication that the message may be a duplicate, because the original message may never have arrived.
I understand that Erlang process message is sync.
When I do
Pid ! message
the sending message thread return right now
the sending message thread will confirm the message has put in the Pid's message queue and then return.
Which choice will the sending message thread do ?
I believe your understanding might be wrong. Erlang message passing is asynchronous.
For example have a look here.
To answer your question then the option number 1 is what's happening here.
In Erlang, message passing is asynchronous. The sender never blocks. Message delivery is not guaranteed. Caveats:
If messaging a local process, then in practice, messages always arrive and do so very quickly.
If messaging a remote process, then messages will be queued for sending. But due to the nature of TCP and distribution, it is not guaranteed that the message will be transferred and processed by the other party.
I think
the sending message thread return right now.
is right.
because Pid ! message just put the message into the message queue of process Pid. the process Pid will use receive to check its message queue. this is nothing with the sending process.