I am fairly new to NServicebus and have run into a problem that I am thinking may have to do with my architecture.
I have one SQS queue with three SNS topics. So let's say:
Queue1
MessageType1
MessageType2
MessageType3
I have created three NServicebus subscribers that will all run as three separate services. Each Subscriber is monitoring Queue1, and each one has a handler for a different message type. This is a rough sketch of how I am envisioning this to work:
---------MessageType3----------
| |
| --MessageType2--------- |
| / | |
V V | |
[Outside Publisher] --MessageType1--> [Queue1] --MessageType1--> [Subscriber1]
| |
| |
/ \
[Subscriber3]<--MessageType3--- ---MessageType2--> [Subscriber2]
An outside service published MessageType1 to the Queue1. Subscriber1 picks up the message, does some processing, and publishes MessageType2 and MessageType3 back to the Queue1. Then Subscribers 2 & 3 pick up their respective messages and do their thing.
But what is happening is it is random which subscriber (1, 2, or 3) picks up the initial MessageType1. So then Sub2 picks it up, and errors because it doesn't have a handler for it.
Why is this happening? I thought the NServicebus would only pick up messages it has a handler for. Does NServicebus only like one message type per queue? Do I need to make separate queues for each message type?
I am hoping there is some way to configure the three subscriber services to only pick up the intended message, but I realize that maybe my understanding of NServicebus is lacking and I need to rethink my design.
Yep, you've got some misconceptions going on here, let's see if I can help clear them up.
What you call "subscribers" are not subscribers. A queue defines a logical endpoint, and multiple processes monitoring that single queue are endpoint instances, not subscribers. They cooperate to scale-out on the processing of a single queue.
A single queue can process multiple message types if you so choose, but when any endpoint instance asks for a message from the queue, it can't control which type it will get. The queue is just a line, it's going to get the message that's next, whatever that message is.
So all the endpoint instances have to have ALL of the message handlers for messages that could go through that queue, or you will get that error.
The only reason to have multiple endpoint instances is for scalability (process more messages at a time) or for availability (process messages on ServerA even if ServerB is getting rebooted.)
Actual subscribers are different. Subscribers are where (in SQS/SNS parlance) a single SNS topic delivers copies of a message to multiple queues. You publish OrderPlaced and one copy goes to the Sales queue just so we can store that a sale was made, and a copy goes to the Billing queue (so that the credit card can be charged) and another copy yet goes to the Warehouse queue (so that they can start the process of getting it ready to put in a box.)
The power of subscribers is that maybe 6 months down the line you create another Subscriber called CustomerCare that subscribes to OrderPlaced in order to store a running total of how much that customer has bought over the past year (see Death to the batch job) and the important bit is you DON'T have to go back to the original code where the order was placed—you just add another subscriber with its own queue.
You might want to check out the NServiceBus step-by-step tutorial which goes over this in a lot of detail.
Related
Let's say on twitter, a celebrity update her status, and it's pushed to all of her followers.
If we set up queue, where the publisher is service that fetches the celebrity's status, and the consumer are the individual followers. But with a million followers, whoever receives that message first will see the update, while others will not ? What is a common pattern to use here so that every one of her followers will see the update and not 'compete' with each other to consume the message first?
I guess you are thinking of queueing system as only having the ability to make point-to-point communication, ie, producer to queue to consumer. This is partly correct. Most queueing systems have atleast two patterns:
Producer-consumer: In this scenario, a message is delivered to just one consumer, ie, if there are multiple consumers, they are competing against each other to get the message from the "queue".
Publish-subscribe: Here, publishers push the message on a "topic" and consumers subscribe to the topic to get messages. The consumers are not competing - each consumer gets all the messages once they have subscribed.
In your example, it's publish-subscribe pattern in action. Underlying implementation may be different, but the basic pattern is that of publish-subscribe.
Refer: https://stackoverflow.com/a/42477769/6886283
I have a series of applications that consume messages from SQS Queues. If for some reason one of these consumers fails and stop consuming messages I'd like to be notified. What's the best way to do this?
Note that some of these queues could only have one message placed into the queue every 2 - 3 days, so waiting for the # of messages in the queue to trigger a notification is not a good option for me.
What I'm looking for is something that can monitor an SQS queue and say "This message has been here for an hour and nothing has processed it ... let someone know."
Possible solution off the top of my head (possibly not the most elegant one) which does not require using CloudWatch at all (according to the comment from OP the required tracking cannot be implemented through CloudWatch alarms). Assume you have the Queue to be processed at Service and the receiving side is implemented through long polling.
Run a Lambda function (say hourly) listening to the Queue and reading messages, however never deleting (Service deletes the messages once processed). On the Queue set the Maximum Receives to any value u want, let's say 3. If Lambda function ran 3 times and all three times message was present in the queue, the message will be pushed to Dead Letter Queue (automatically if the redrive policy is set). Whenever new message is pushed to dead letter queue, it is a good indicator that your service is either down or not handling the requests fast enough. All variables can be changed to suit your needs
I am looking to run a service that will be consuming messages that are placed into an SQS queue. What is the best way to structure the consumer application?
One thought would be to create a bunch of threads or processes that run this:
def run(q, delete_on_error=False):
while True:
try:
m = q.read(VISIBILITY_TIMEOUT, wait_time_seconds=MAX_WAIT_TIME_SECONDS)
if m is not None:
try:
process(m.id, m.get_body())
except TransientError:
continue
except Exception as ex:
log_exception(ex)
if not delete_on_error:
continue
q.delete_message(m)
except StopIteration:
break
except socket.gaierror:
continue
Am I missing anything else important? What other exceptions do I have to guard against in the queue read and delete calls? How do others run these consumers?
I did find this project, but it seems stalled and has some issues.
I am leaning toward separate processes rather than threads to avoid the the GIL. Is there some container process that can be used to launch and monitor these separate running processes?
There are a few things:
The SQS API allows you to receive more than one message with a single API call (up to 10 messages, or up to 256k worth of messages, whichever limit is hit first). Taking advantage of this feature allows you to reduce costs, since you are charged per API call. It looks like you're using the boto library - have a look at get_messages.
In your code right now, if processing a message fails due to a transient error, the message won't be able to be processed again until the visibility timeout expires. You might want to consider returning the message to the queue straight away. You can do this by calling change_visibility with 0 on that message. The message will then be available for processing straight away. (It might seem that if you do this then the visibility timeout will be permanently changed on that message - this is actually not the case. The AWS docs state that "the visibility timeout for the message the next time it is received reverts to the original timeout value". See the docs for more information.)
If you're after an example of a robust SQS message consumer, you might want to check out NServiceBus.AmazonSQS (of which I am the author). (C# - sorry, I couldn't find any python examples.)
Iam sending messages to the queue and using amazon sqs queuing system in a rails application. But since the queue follows FIFO process, it will get the next items in the same fashion. Suppose if I have 100 items in a queue, how can I retrieve the 35th item from the queue and process it. As far as I know, there is no such method that amazon sqs provides for doing it. So is there any other method/workaround where I can achieve the this functionality.
There is no method to do that; SQS does not guarantee order of items in the queue due to its geographically redundant nature; it can't even guarantee FIFO. If you absolutely must process things in order, and need the ability to 'look ahead' in the queue, SQS may not be your best choice. Perhaps a custom made queue in something like DynamoDB may be work better.
SQS is designed to guarantee at-least-once delivery and does not take into account the order of messages. So the simple answer to your question on whether you can do that, is no.
A work around would depend on your use-case:
To split work among different processes handling queue messages and making sure they don't both process the same item - Different queues is one approach, or prefixing every message with an identifier denoting which process is supposed to work on it. For example, if I have 4 daemons's running, I could prefix every message in the queue with the ID of the process which should work on it - 1,2,3 or 4. Every process would only process messages with the number corresponding to it's ID.
Order of arrival is critical - In this case, you're better off not using SQS because it wasn't to be used this way. CloudAMQP is a cloud based service that is based off RabbitMQ which is a true FIFO queue and would suit this case better than SQS.
I am going to make the rails application which integrates the Amazon's cloud services.
I have explore amazon's SNS service which gives the facility of public subscription which i don't want to do. I want to notify only particular subscriber.
For example if I have 5 subscriber in one topic then the notification should be goes to particular subscriber.
I have also explored amazon's SQS in which i have to write a poller which monitor the queue for message. SQS has also a lock mechanism but the problem is that it is distributed so there would be a chance of getting same message from another copy of queue for process.
I want to know that what would be the possible approach to go.
SQS sounds like what you want.
You can run multiple "worker" processes that compete over messages in the queue. Each message is only consumed once. The logic behind the "lock" / timeout that you mention is as follows: if one of your workers were to die after downloading a message, but before processing it, then you want that message to eventually time out and be re-downloaded for processing on another node.
Yes, SQS is built on a polling model. For example, I have a number of use cases in which I use a minutely cron job to poll for new messages in the queue and take action on any messages found. This pattern is stupid simple to build and works wonders for a bunch of use cases -- a handy little "client" script that pushes a message into the queue, and the cron activated script that will process that message within a minute or so.
If your message pattern is extremely sparse -- eg, only a few messages a day -- it may seem wasteful to poll constantly while the queue is empty. It hardly matters.
My original calculation was that a minutely cron job would cost $0.04 (now $0.02) per month. Since then, SQS added a "Long-Polling" feature that lets you achieve sub-second latency on processing new messages by sending 1 "long-poll" message every 20 seconds to poll an idle queue. Plus, they dropped the price 50%. So per month, that's 131k messages (~$0.06), a little bit more expensive, but with near realtime request processing.
Keep in mind that a minutely cron job I described only costs ~$0.04 / month in request load (30d*24h*60m * 1c / 10k msgs). So at a minutely clip, cost shouldn't really be a concern here. Even polling every second, the price rises only to $2.59 / mo, not exactly a bank buster.
However, it is possible to avoid frequent polling using a webservice that takes an SNS HTTP message. Such an architecture would work as follows: client pushes message to SNS, which pushes message to SQS and routes an HTTP request to your webservice, triggering it to drain the queue. You'd still want to poll the queue hourly or daily, just in case an HTTP request was dropped. In the end though, I'm not sure I can think of any scenario which really justifies such complexity. I'd much rather pay $0.04 a month to have a dirt simple cron job polling my queue.