I'm trying to create a reasonable setup for client-client-communication for our existing infrastructure. I've been reading the docs for Spring, Websocket, STOMP, SockJS and ActiveMQ for over a week now and I'm not not sure whether what I am trying to do is feasible or sensible. Spring server and JavaScript client were up and running relatively quickly and sending messages between clients just works (direct connection from JS client to Spring server). This setup won't suffice for our needs so we decided to put dedicated brokers in between. Configuring ActiveMQ is a nightmare probably because I don't really know where to start. I have not worked with a dedicated broker so far.
Environment
170 independent servers (Tomcat, Spring, SockJS, STOMP)
2 ActiveMQ (Artemis) brokers (load balance, failure safety)
a few thousand clients (JavaScript/.NET, SockJS, STOMP)
Requirement
I need every client to be able to talk to every other client. Every message has to be curated by one of the servers. I'd like the clients to connect to one of the ActiveMQ brokers. The ActiveMQ brokers would hold a single connection to every single server. The point is to avoid that all my clients would have to open 170 WebSocket connections to all the servers. The servers do not need to talk to each other (yet/necessarily) since they are independent with different responsibilities.
Question
Is ActiveMQ or any other dedicated broker viable as transparent proxy/relay i.e. can it handle this situation and are there ways to dynamically decide the correct recipients or should I go another route like rolling my own Spring-based relay?
In a traditional messaging use-case (e.g. using ActiveMQ Artemis with STOMP) the broker manages "destinations" and any messages sent to those destinations. Messages are only dispatched to clients if the client specifically creates a consumer on a destination.
In your use-case all of your 170 "servers" would actually be messaging clients. They would need to create a consumer on the broker in order to receive messages. To be clear, once the consumer is created then the broker would dispatch messages to it as soon as they arrived.
I'm not sure what exactly you mean by "transparent," but if it means that the process(es) receiving the message(s) don't have to do anything then no message broker will handle your use-case.
Related
I have the doubt if a MQTT Broker also can be a client? Or I must need to separate and get a dispositive to act as a broker and another as a client. I'm not finding information on internet.
If I understand the question properly, what you are looking for is MQTT bridging.
This is where one broker acts as a client to a second (or multiple) broker and based on config copies messages on topics between the 2 brokers.
How this is configured depends on which broker you are using, but the concept is part of the spec (I don't think it's optional off the top of my head) so all brokers should support it
I am willing to create a mediator which is subscribed and published to 2 separate broker who have no access to topics of each other. The aim is to updates and create a logic of the message published by broker 1 and send it to broker 2 according to the set of rules
Do I need 2 separate ports ? As the topic level might be different in both brokers
Any help is much appreciated!!!
There is no MQTT standard (note, can only speak for 3.1.1) defined features that would allow a client maintain two concurrent connections. Therefore, this is entirely broker implementation-dependent and necessitates a bridge.
For example, the Eclipse Mosquitto broker can be configured as a bridge to another broker and even remap topics from itself to a different topic structure of the other. Please refer to the Mosquitto man page section Configuring Bridges for the specifics.
As far as creating a bespoke application, you can always write a simple Python program that is running two instances of an MQTT client (Eclipse Paho for example, which has a lightweight asyncio wrapper to facilitate concurrency), each connected to different brokers. The glue logic between them just has to re-publish an incoming subscribed topic message from Broker A to some topic, with or without a remapping step, to Broker B.
If the two brokers are both running locally on a single NIC, then you would need to use different ports.
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.
We are implementing an MQ/IIB architecture where we will have one QM and one Broker each on 2 RHEL servers load-balanced with each other to divide incoming traffic.
We have consumer applications which connect our servers through JMS bindings file. We also have IIB applications running on both of them.
Now, since one bindings file could have only one QMGR name while creating a connection factory, it's not recommended to keep different QM/Broker names on each servers. Since this bindings file would be shared with consumers, it has to be with unique QM name.
But if we have same QM/Broker names on each server, all logs on IIB record and replay tool will have one Broker name (from both servers) which is again difficult to identify which server actually served the incoming request.
Could you please suggest best possible approach in such scenario?
Or else suggest if above approach can be modified to achieve our goal.
In general it is not a good practice to have two queue managers with the same name. The same would be true for IIB brokers for the reasons you stated.
In the Binding file you can leave QMANAGER blank (null). This will allow the application to connect to any queue manager listening on the HOSTNAME and PORT that you specify.
If the queue managers on the 2 RHEL servers use the same port you could even set hostname to localhost and use the same binding file on both servers.
Example is below if both queue managers listened on the same port:
DEFINE CF(CF_NAME) QMANAGER() TRANSPORT(CLIENT) CHANNEL(MY.SVRCONN) HOSTNAME(localhost) PORT(1414)