Tibco stores messages in a data file or in database. I need to browse old messages posted to a topic or queue. Does EMS provide any API for searching old/consumed messages?
Suppose I have a topic complexEvent.topic at time t1 a publisher publishes a message to complexEvent.topic, at time t2 a consumer consumes the message. Say after 2 hours from t2 I need to view all messages posted between t1 and t1+ 10 (minutes) to topic complexEvent.topic.
How can I search old messages?
EMS does not store copies of messages by design : A MOM is not a DBMS, and a MOM should normally be optimized for performance.
For a SPECIFIC and PUNCTUAL need such as yours, I would simply create an EMS bridge (Similar to MQ Alias, see EMS doc) from the Destination(Topic,Queue) you want to log to a "DESTINATIONNAME.log" QUEUE. This way, the current code and destination are unaffected.
After that, you could use a Java QueueBrowser and MessageSelector to search for messages in your log queue.
See this Oracle Documentation for details.
Don't forget to clean up the queue (You could also set up a message limit and overflow strategy).
If performances are critical, consider storing the logging queues on different EMS instances.
Related
Would the mqtt community consider placing message information in the topic name an anti-pattern?
I have a client that has a vast library written around rabbitmq, and I'm trying to tweak their client and server code to allow them to configure their services for mosquitto instead. One central requirement for them is TTL, the clients can sometimes sit for hours publishing data before the server comes back online and they do not want messages to show up that are beyond their TTL.
Their message envelope system is an elaborate json and 1) it would be painful to wrap or alter this json 2) I do not want to incur the expense of unmarshalling json to retrieve a timestamp.
The easiest thing to do is place the timestamp at the end of the topic and consume with wildcards: mytopic/mysubtopic/{timestamp} consumed by mytopic/mysubtopic/#
Are there any unintended consequences for this, and would this be considered an anti-pattern?
Whether this is an anti-pattern is a matter of opinion; the spec defines the topic as "The label attached to an Application Message..." so does not preclude your usage. I can think of a few potential "unintended consequences" to your approach (which may, or may not, apply to your specific situation):
Retain flag: As per your comment you will not be able to set the Retain flag to 1 (because all messages would be retained).
Latest Message only when comms re-established: A subscriber may only want the latest message when communications are re-established. This can be achieved by publishing messages with the retain flag set to 1 which results in your subscriber receiving the latest message (and only the latest message; subject to QOS/CleanSession) on each topic it subscribes to (docs). As per the above this will not work with your topic structure.
Order of delivery: the spec requires that "A Server MUST by default treat each Topic as an 'Ordered Topic'" but there is no such guarantee across topics. Note that ordered delivery is dependent upon settings (see the "Non normative comment" in the spec) so this may not be an issue.
Topic Alias: MQTT V5 introduces Topic Alias which can be used to reduce the amount of data transmitted. This will not provide a benefit with your structure.
I am using rabbitmq to communicate between microservices written in ruby on rails. Each service subscribes to a topic. All services are scaled and run as multiple instances based on need.
During subscription bunny moves all the messages from the queue into unacked state. This makes other scaled instances to be just idle, since there is no message in ready state.
Is there a way to limit the number of messages a subscription can fetch, so that other instances can take the remaining messages from the queue.
Based on the information you made available, I'm assuming you're using rubybunny. If this assumption is incorrect (there are other ruby clients available for rabbitmq) let me know and/or check the documentation related to your client.
Back to rubybunny, link provided points to necessary information, quoting it:
For cases when multiple consumers share a queue, it is useful to be
able to specify how many messages each consumer can be sent at once
before sending the next acknowledgement.
In AMQP 0.9.1 parlance this is known as QoS or message prefetching.
Prefetching is configured on a per-channel basis.
To configure prefetching use the Bunny::Channel#prefetch method like so:
ch1 = connection1.create_channel
ch1.prefetch(10)
Does tibco support "multicast" ?
I guess another term used is "worker queues". (as seen in the rabbitmq link below)
See : http://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html
I call them "fighters", as in, several processes can be wired to one queue, and when a message arrives in the queue, ONE of the several processes will get the message,but not all of them.
In EMS and most JMS based messaging system (supporting Queues and Topics), this is ALREADY the default behavior.
In would not call that "multicast" or "worker queues", but simply "load sharing" or "load balancing". Active-Mq calls it "Clustering" (I don't like the term, but the diagram is neat).
The official name for the pattern is "Competing consumers (EIP)".
Whatever you call it, it's super easy to do in EMS. By default, queue accept multiple clients for reading (you can change this and make them exclusive, see the user doc). When a queue is read by 2 or more consumers, and a message is sent to the queue, the message will go to one of ANY consumers. Hence your expected behavior.
Please refer to the same link for another chapter (14, page 411) on "Multicast" with EMS. This is different... it's ACTUAL NETWORK BASED Multicast, meant for helping lowering network traffic when a topic does publications to a massive amount of subscribers.
FYI, EMS is only one out of three messaging solution from TIBCO. The other two are Rendez-vous(older, UDP based) and FTL (newer, low latency solution).
This is newbie question I'm sure, but using Mosquitto how do I subscribe to a retained message and guarantee that I only get the message once? I have a Paho based java subscriber, and in testing what I notice is, if my subscriber is brought down, and then restarted it pulls some of the same messages that it received before it came down. (Normally it would stay up forever, but stuff happens.) And it's critical the subscriber get each message only once. (For my use case there is only 1 subscriber.)
Is there something built into Mosquitto that provides this mechanism or is this something I have to build into my data schema and retrieval process?
I see there is a way of deleting a message by topic (https://lists.launchpad.net/mosquitto-users/msg00067.html) but right now my topics are pretty generic [Company]/[MAC address]/[Topic x] and it's possible to have two separate messages on the queue with the same topic, and I'm assuming if I send an empty message to a given topic I'll end up deleting all messages of said topic.
Thanks in advance for any help on this.
set the QoS level as 2.
QoS-level-2:
Exactly-once delivery.
This is the highest level that also incurs most over head interms of control messages and the need for locally storing the messages.
Exactly-once is a combination of at-least-once and at-most-once delivery guarantee.
Based on this answer here, I need to put emails in a queue and have a background task run and send them. How do I do this with an architecture that is of ASP.NET-MVC and WCF?
How do I build a queue (sql server)?
How do I build a background task?
You can skin this cat many different ways. The key being that the actual sending of the emails is asynchronous to the queuing of the email.
Queue messages via WCF Service using MSMQ binding via this series of blog posts, which assumes IIS 7: MSMQ, WCF, and IIS: Getting Them to Play Nice.
Queue messages to MSMQ. MSMQ is a nice (sometimes underutilized) queue service built into Windows. You'll write a Windows service to receive messages from this queue. If you have IIS 7, then check out Death to Windows Services, Long Live AppFabric. MSMQ is a breeze, but has some quirky constraints (4MB message size and availability)
Queue messages to a 'sql queue'. Create a table to hold basic queued message information and then stored procedures to wrap the queue semantics (e.g. you don't want multiple consumers to receive the same message). Not difficult, but a little time consuming to get right.
Queue messages to Service Broker (or even MSMQ) and write a Windows service that receives messages from the Service Broker Queue. Service Broker handles the queueing semantics (competing consumers) for you. The downside is that its a pain in the ass to administer.
HTH,
Z
I think your solution is independant of the fact you're using MVC.
The way I've implemented this in the past is to persist the fact you need to sent an e-mail into the database and then process this using a Windows Service.
Another way to do this would be to utilize MSMQ as your storage medium. In general, MSMQ shouldn't be used to "store" data, only as a message transport mechanism, but it's certainly an option in this case.
In terms of developing a "queue", if the e-mails need ordered delivery for some reason, simply having a "RequestedDTTM" column in your database table would allow you to send them in the order they were requested.
Lastly, I would consider implementing a simply multi-threaded e-mail sender to maximize performance. Using the TPL in .NET 4.0 would make this pretty easy. Alternatively, you could use something like the SmartThreadPool library (available at codeplex.com) to manager your e-mail sender threads.
As was mentioned in the other answer you linked to, your UI shouldn't be doing this e-mail sending.