I'm trying to build a serverless async polling service that triggers an async data request, waits for it to finish (by trying periodically) and then polling the response once its ready.
Ideally, I would trigger the data request with Lambda and push an event for EventBridge to later be processed by another Lambda that queries the API to see if the reseponse is ready, and if not - push a new event to EventBridge to try again later. For that to work, I need a way to make EventBridge wait before it forwards the event onwards. I'm trying to come up with a solution that doesn't require Lambda idle time.
Is there a way to make EventBridge wait on an event before it is pushed onwards? Or alternatively to setup a one-time scheduled event?
Sounds like the newly announced EventBridge scheduled events would solve your problem! You could simply schedule your event for Date.now() + x seconds or similar.
Announcement blog post: https://aws.amazon.com/blogs/compute/introducing-amazon-eventbridge-scheduler/
AWS Docs: https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html
Amazon SQS delay queues provides that functionality:
Delay queues let you postpone the delivery of new messages to a queue
for a number of seconds, for example, when your consumer application
needs additional time to process messages.
Related
I've configured and AWS EventBridge schedule that triggers a lambda target every hour.
How can I best test this trigger without having to wait until the schedule is reaching 1 hour?
Sometime things are simple....
The Lambda interface actually provides a Test function with with you can generate events:
I guess I looked in the wrong place to generate events (in the eventbridge itself).
I have a situation where I have a worker that makes multiple calls to an external API. The problem is that we have a threshold of many calls we can make to this API per hour.
What I'd like to do is to create a worker which will make these many sequential calls to this external API. If in between these calls we get an error because we've reached the number of connections we're allowed in that hour, the worker would then save the document and schedule a new worker to complete the remaining API calls at a later time (maybe 1, 2 hours later. Ideally this should be configurable e.g.: 10mins, 1hour, etc).
Is there someway I could achieve this?
With SideKiq you can scheduled when a job will be executed with a friendly API :
MyWorker.perform_in(3.hours, 'mike', 1) # Expect a duration
MyWorker.perform_at(3.hours.from_now, 'mike', 1) # Expect a date
Check it out : Scheduled Jobs
You want Sidekiq Enterprise and its Rate Limiting API. The alternative is tracking the rate limit yourself and rescheduling the job manually.
https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting
I'm using sidekiq pro for my application, and it's been working great. But I'd like to have a way to notify my users that a failed job is being retried.
A flow would go something like this:
Batch starts
worker1 runs successfully
worker2 runs successfully
worker3 fails
oncomplete fires, stuff happens
worker3 restarts
** onretry fires, notification sent to user
worker 3 runs successfully
onsuccess fires, stuff happens
My imaginary onretry doesn't exist in the documentation but I'm hoping there's a way to fake it. I know that I can tell if the batch has failures via the status object, but I don't see a way to get a retry event. Is there such a thing?
The most workable approach is likely a server-side middleware which can detect a retry in progress for a batched job and send an email.
I have a series of applications that consume messages from SQS Queues. If for some reason one of these consumers fails and stop consuming messages I'd like to be notified. What's the best way to do this?
Note that some of these queues could only have one message placed into the queue every 2 - 3 days, so waiting for the # of messages in the queue to trigger a notification is not a good option for me.
What I'm looking for is something that can monitor an SQS queue and say "This message has been here for an hour and nothing has processed it ... let someone know."
Possible solution off the top of my head (possibly not the most elegant one) which does not require using CloudWatch at all (according to the comment from OP the required tracking cannot be implemented through CloudWatch alarms). Assume you have the Queue to be processed at Service and the receiving side is implemented through long polling.
Run a Lambda function (say hourly) listening to the Queue and reading messages, however never deleting (Service deletes the messages once processed). On the Queue set the Maximum Receives to any value u want, let's say 3. If Lambda function ran 3 times and all three times message was present in the queue, the message will be pushed to Dead Letter Queue (automatically if the redrive policy is set). Whenever new message is pushed to dead letter queue, it is a good indicator that your service is either down or not handling the requests fast enough. All variables can be changed to suit your needs
I am going to make the rails application which integrates the Amazon's cloud services.
I have explore amazon's SNS service which gives the facility of public subscription which i don't want to do. I want to notify only particular subscriber.
For example if I have 5 subscriber in one topic then the notification should be goes to particular subscriber.
I have also explored amazon's SQS in which i have to write a poller which monitor the queue for message. SQS has also a lock mechanism but the problem is that it is distributed so there would be a chance of getting same message from another copy of queue for process.
I want to know that what would be the possible approach to go.
SQS sounds like what you want.
You can run multiple "worker" processes that compete over messages in the queue. Each message is only consumed once. The logic behind the "lock" / timeout that you mention is as follows: if one of your workers were to die after downloading a message, but before processing it, then you want that message to eventually time out and be re-downloaded for processing on another node.
Yes, SQS is built on a polling model. For example, I have a number of use cases in which I use a minutely cron job to poll for new messages in the queue and take action on any messages found. This pattern is stupid simple to build and works wonders for a bunch of use cases -- a handy little "client" script that pushes a message into the queue, and the cron activated script that will process that message within a minute or so.
If your message pattern is extremely sparse -- eg, only a few messages a day -- it may seem wasteful to poll constantly while the queue is empty. It hardly matters.
My original calculation was that a minutely cron job would cost $0.04 (now $0.02) per month. Since then, SQS added a "Long-Polling" feature that lets you achieve sub-second latency on processing new messages by sending 1 "long-poll" message every 20 seconds to poll an idle queue. Plus, they dropped the price 50%. So per month, that's 131k messages (~$0.06), a little bit more expensive, but with near realtime request processing.
Keep in mind that a minutely cron job I described only costs ~$0.04 / month in request load (30d*24h*60m * 1c / 10k msgs). So at a minutely clip, cost shouldn't really be a concern here. Even polling every second, the price rises only to $2.59 / mo, not exactly a bank buster.
However, it is possible to avoid frequent polling using a webservice that takes an SNS HTTP message. Such an architecture would work as follows: client pushes message to SNS, which pushes message to SQS and routes an HTTP request to your webservice, triggering it to drain the queue. You'd still want to poll the queue hourly or daily, just in case an HTTP request was dropped. In the end though, I'm not sure I can think of any scenario which really justifies such complexity. I'd much rather pay $0.04 a month to have a dirt simple cron job polling my queue.