Is there any documentation on what the subsections mean under SolAdmin -> Statistics -> Client Statistics
There are two sections Total Ingress Discards / Total Egress Discards and further classification under that. I would like to understand more on the subsections under ingress/egress discards
What does Ingress and Egress messages mean with respect to solace? Is it incoming/outgoing messages from a message broker?
Any pointers to documentation on the subsections under Ingress/Egress Discards?
What does Message Spool Ingress Discard mean?
Tried searching the documentation and there was not much help. Appreciate if you can point me to the right documentation or share your knowledge here.
thanks
Yes, the ingress/egress are incoming/outgoing from the broker's perspective, that means Ingress are messages sent TO Solace from the client and egress are messages sent FROM Solace to the client.
You can find description for every discard reason on this link, on the "Description of Detailed Spooled Message Statistics" section:
Solace Message Spool Information
In general Message Spool Ingress discard means that the broker could not receive and store your message properly on the Message Spool (Guaranteed Messaging), and the reason would be one of the list that you can see in the link of Item 2.
Hope this helps.
Related
According to the MQTT specification, a QoS 2 message sent by a MQTT client must follow this workflow:
During the various phases Mosquitto stores the message in its memory. This is also confirmed by looking at the mosquitto.db persistent storage using the db_dump tool described here.
The question is: if a malicious client PUBLISH tons of messages with QoS 2 but never sends the PUBREL message as a response to PUBREC what happens ? Mosquitto keep the messages undefinitively ? I expected some kind of configuration parameter able to get rid of such unacknowledged messages after some time but I can't find any.
I'm not 100% sure, but I think the max_inflight_messages setting should kick in here and not allow the client to send the second QOS 2 message until the first has been completed.
This would limit each client to one malicious message at time.
There are a number of libraries that will allow you low level control over when packets are sent so building a PoC shouldn't be that hard. And if it is possible to trigger a DoS style attack, I'm sure eclipse/mosquitto would look kindly at a Pull Request with a fix.
I am working on pipeline where JMS is sending messages to Solace queues and streaming job is consuming data from solace queues.
Here i want to plot Ingress/Egress message count on graph.
But there is huge difference between count.
here are message spool stats:
Number of ingress messages - 159479012
Number of egress messages - 222689956
How this is possible that egress count is higher?
Without knowing all the details, one scenario that I can think of which can lead to a higher egress count compared to ingress is when you have more consumers than publishers. For example, if you have a publisher publishing messages on a topic and 10 consumers with 10 queues subscribing to that topic, that message will be delivered 10 times and hence, your egress count will be 10x than the ingress count.
Additionally, if you have messages being redelivered, they would also count towards your egress.
In case, this doesn't answer your question, you might want to post this question on Solace Community (https://solace.community/).
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
This is newbie question I'm sure, but using Mosquitto how do I subscribe to a retained message and guarantee that I only get the message once? I have a Paho based java subscriber, and in testing what I notice is, if my subscriber is brought down, and then restarted it pulls some of the same messages that it received before it came down. (Normally it would stay up forever, but stuff happens.) And it's critical the subscriber get each message only once. (For my use case there is only 1 subscriber.)
Is there something built into Mosquitto that provides this mechanism or is this something I have to build into my data schema and retrieval process?
I see there is a way of deleting a message by topic (https://lists.launchpad.net/mosquitto-users/msg00067.html) but right now my topics are pretty generic [Company]/[MAC address]/[Topic x] and it's possible to have two separate messages on the queue with the same topic, and I'm assuming if I send an empty message to a given topic I'll end up deleting all messages of said topic.
Thanks in advance for any help on this.
set the QoS level as 2.
QoS-level-2:
Exactly-once delivery.
This is the highest level that also incurs most over head interms of control messages and the need for locally storing the messages.
Exactly-once is a combination of at-least-once and at-most-once delivery guarantee.
With an MQTT broker, is it possible to set up multiple consumers for a topic such that for any given message on that topic only one consumer will receive the message?
The short answer is no, not with any broker that purely implements the MQTT spec.
I suppose it would be possible to write a broker that talked to the clients using MQTT and only delivered messages to a single subscriber. (It would have to deliver with QOS2 to ensure that every message was consumed)
By coincidence I was talking to a colleague about something similar to this sort of thing earlier in the week, he had found a way to do it using IBM* MQ Light and something called 'Shared Destinations'. (MQ Light uses AMPQ not MQTT)
https://developer.ibm.com/messaging/mq-light/
full disclosure, I work for IBM
UPDATE:
I've since been informed that the IBM MessageSight v1.2 appliance can actually do shared destinations using MQTT (http://www-03.ibm.com/software/products/en/messagesight)
UPDATE 2:
Shared subscriptions is an optional part of the MQTT v5 spec so worth checking any v5 brokers for the option.
Look at Shared Subscriptions https://issues.oasis-open.org/browse/MQTT-234
some MQTT servers support it.
EMQTT (open source):
https://github.com/emqtt/emqttd/issues/639#issuecomment-247851593
HiveMQ:
http://www.hivemq.com/blog/mqtt-client-load-balancing-with-shared-subscriptions/
IBM MessageSight:
http://www.ibm.com/support/knowledgecenter/SSCGGQ_1.2.0/com.ibm.ism.doc/Developing/devsharedsubscriptions.html
VerneMQ:
https://vernemq.com/docs/configuration/balancing.html
That is not possible. In MQTT all subscribers to a particular topic receive messages published to said topic. In order to direct a message to a particular subscriber, both publisher and subscriber would have to use a particular topic different to that used by other subscribers.
Independent of the broker that you're are using, you can use Apache Camel to implement a route that copies all messages from Topic A to Topic B.
Or copy only specific messages that match an specific rule such as user, content pattern, QoS.
Other solution is using a multi-protocol broker such as ActiveMQ and copy specific message topics to a Queue (queues only can have one consumer) and consume the queue with another protocol such as JMS or STOMP.