Is it possible for me to get a message from the SQS queue based on the message ID with the Amazon PHP SDK? Do I have to just grab all of the messages on the queue and then filter it on my server?
My server receives a SNS instigated request with a queue message Id and I'm having to filter the message from an array of messages from SQS.
The purpose of a queue is to use it as a buffer to store messages before a certain processing task. This should not be confused with a storage service or a database service. Amazon SQS allows a process to retrieve messages from a queue (buffer) and process them as needed. If needed Standard or FIFO queues can be used.
In answer to your question: SQS does not provide a mechanism to retrieve by Message ID. So as you suggested, you can have separate workers to retrieve all messages in parallel and look for the message with the ID you want. (This can be exhaustive)
Since your use case is similar to that of a storage service, I suggest writing to a storage service and retrieving from it based on a column named "Message ID".
Just so that it may help someone else. I couldn't find a straight forward method to do this. Instead, implementing another set of queue workers to delegate the tasks solved the performance issue. These workers performed only two tasks
Retrieve queue item
Associative valid id
Send it to the processing server (after performing load checks, availability etc)
I would have to know more about your use case to know, but it sounds like you are using SQS as a database. Might I recommend instead of sending messages to SQS and sending the message ID to SNS, instead adding a row in DynamoDB, and sending to key through SNS?
Or even better yet, just send the raw data through SNS?
Related
AWs recently added a feature that allows you to send messages from a DLQ back to source queue via a lick of a button "redrive to source". I wanted to know is this possible via an API call.
I know how to extract a message from dlq queue and re send it, But with this new function i was hoping i wouldnt need to handle the messages, but rather just call a method perhaps on the queue and if its configured it would do the redelivery.
Anyone know if this is possible, as im searching in the net.
I believe currently this feature is only available via the management console UI and not as an API
I have an SNS topic called onboard-user. A lot of microservices subscribe to this topic
for ex:
Emai microservice - to send the email
System preferences microservice -To set the defaults
There is a queue in these microservices that subscribes to the onboard-user SNS topic.
Now, if the email to the user needs to be resent, how should it be handled? onboard-user should not get a new message in that case as that has its own purpose and we would not want all of the microservices listening to it to be notified.
Having an alternate SNS topic for send emails doesn't sound like such a good idea and an overkill. what's the right way to handle it?
Why would an email need to be resent? Due to a transient error or due to some business logic initiated by an end user? The solution depends on the problem. I would argue that if you detect the case where a message needs to be recent that you either: (A) maintain separate resend email SQS queue OR (B) put a new message on the existing email queue with a field indicating that this is a resend.
Since SQS is pretty cheap, I could lean towards (A).
If my understanding is correct, you have single SNS topic to do multiple processing and that's why multiple microservices are listening to it. Please correct, if that is incorrect.
If that is the case, the better option is to:
a. Segregate email processing - keep a separate SNS topic for email purpose e.g. 'inform-user' and use it for retrying
b. If you can't segregate, then the only option in above scenario is to have "state" in the message itself. That means having an attribute in the message to indicate that it is for "retrying" and then the listener microservice should process accordingly.
I went through the AWS pinpoint documentation for checking delivery status of sent SMS message. But, I couldn't find any batter way than streaming SMS Event to either Amazon Kinesis Data Streams or Amazon Kinesis Data Firehose then sending data to Amazon S3 bucket or Amazon Redshift database and only after that to the application.
Here, my requirement is to send just verification code and get the delivery status of SMS message. Is there any better way of achieving this? Because the way I mentioned above is too lengthy and costly. Any help will be appreciated.
sendMessages() API returns status of SMS. It is assumed that this API may not instantly return send status as SMS may not get delivered instantly.
But since "callback", it takes time to return, works Asynchronously.
What is the status it is returning?
refer to: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Pinpoint.html#sendMessages-property
In callback, json "data" returned has "MessageResponse" has field "Result" has field "DeliveryStatus" which has possible values as below:
"SUCCESSFUL"
"THROTTLED"
"TEMPORARY_FAILURE"
"PERMANENT_FAILURE"
"UNKNOWN_FAILURE"
"OPT_OUT"
"DUPLICATE"
Why is status update through Kinesis, then S3 buckets, then Analytics is needed? Actually not needed. With Kinesis you get more detailed information.
On hard pressing simpler ways, AWS expert mentioned use Lamda service to directly send to some "Queue" (possibly another AMZ product) or call HTTP end point through Lamda service. All these services are costly and cost other resources too. "Total Cost of Ownership" just grows or these guys are "up-selling" more products.
I want to send transactional SMS. That's it. I need to know if SMS was delivered or not. That's it. But you end spending for Kinesis (how ever small or large cost), for S3, for Analytics Services, associated EC2, EBS everything.
I discussed hard with AMZ Architect, who settled to, with Kinesis you can directly poll and get status if you configure Pinpoint with Kinesis. So I get SMS event updates ALSO through Kinesis, which are "similar" to what you get by calling getMessages() API.
In SMS event stream directly through Kinesis, you get
"event_type" value as
"_SMS.BUFFERED"
"_SMS.SUCCESS"
"_SMS.FAILURE"
"_SMS.OPTOUT"
reference:
https://docs.aws.amazon.com/pinpoint/latest/developerguide/event-streams-data-sms.html
And value of "record_status" field you get
"SUCCESSFUL"
"DELIVERED"
"PENDING"
"INVALID"
"UNREACHABLE"
"UNKNOWN"
"BLOCKED"
"CARRIER_UNREACHABLE"
"SPAM"
"INVALID_MESSAGE"
"CARRIER_BLOCKED"
"TTL_EXPIRED"
"MAX_PRICE_EXCEEDED"
Some of these status code may be irrelevant to your case. e.g. "TTL_EXPIRED" is possible value if you are sending "delayed" SMS.
Kinesis, delivered more detailed, but may not be practically useful.
If you exceed your max SMS cost Quota with Kinesis, you get the reason. With sendMessages() API you won't. But you can check SMS quota through APIs.
refer to: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ServiceQuotas.html#getServiceQuota-property
I strongly believe, if sending "Transactional" SMS which are by nature priority delivered, you don't need Kinesis.
Amazon is overselling by entangling services into composite solutions, in some cases. Anyways, they are not wrong, Pinpoint is "Marketing" product to do Campaigns, User Journey and Conversion Analysis. Which is bigger business case for Amazon that "just sending SMS". You may not get best advise for your Usecase, so negotiate hard to save your $s.
Preamble: I'm trying to put together a proposal for what I assume to be a very common use-case, and I'd like to use Amazon's SWF and SQS to accomplish my goals. There may be other services that will better match what I'm trying to do, so if you have suggestions please feel free to throw them out.
Problem: The need at its most basic is for a client (mobile device, web server, etc.) to post a message that will be processed asynchronously without a response to the client - very basic.
The intended implementation is to for the client to post a message to a pre-determined SQS queue. At that point, the client is done. We would also have a defined SWF workflow responsible for picking up the message off the queue and (after some manipulation) placing it in a Dynamo DB - again, all fairly straightforward.
What I can't seem to figure out though, is how to trigger the workflow to start. From what I've been reading a workflow isn't meant to be an indefinite process. It has a start, a middle, and an end. According to the SWF documentation, a workflow can run for no longer than a year (Setting Timeout Values in SWF).
So, my question is: If I assume that a workflow represents one message-processing flow, how can I start the workflow whenever a message is posted to the SQS?
Caveat: I've looked into using SNS instead of SQS as well. This would allow me to run a server that could subscribe to SNS, and then start the workflow whenever a notification is posted. That is certainly one solution, but I'd like to avoid setting up a server for a single web service which I would then have to manage / scale according to the number of messages being processed. The reason I'm looking into using SQS/SWF in the first place is to have an auto-scaling system that I don't have to worry about.
Thank you in advance.
I would create a worker process that listens to the SQS queue. Upon receiving a message it calls into SWF API to start a workflow execution. The workflow execution id should be generated based on the message content to ensure that duplicated messages do not result in duplicated workflows.
You can use AWS Lambda for this purpose . A lambda function will be invoked by SQS event and therefore you don't have to write a queue poller explicitly . The lambda function could then make a post request to SWF to initiate the workflow
Following a specific action the user takes on my website, a number of messages must be sent to different emails. Is it possible to have a separate thread or worker take care of sending multiple emails so as to avoid having the response from the server take a while to return if there are a lot of emails to send?
I would like to avoid using system process or scheduled tasks, email queues.
You can definitely spawn off a background thread in your controller to handle the emails asynchronously.
I know you want to avoid queues, but another thing i have done in the past is written a windows service that pulls email from a DB queue and processes it at certain intervals. This way you can separate the 2 applications if there is a lot of email to be sent.
This can be done in many different ways, depending on how large your application is and what kind of reliability you want. Any of these ways should help you achieve what you want (in ascending order based on complexity):
If you're using IIS SMTP Server or another mail server that supports a pickup directory option, you can go with that. With this option, instead of sending the emails directly, they are saved first in the pickup directory. Your call will immediately return after the email is saved in the pickup directory, so the user won't have to wait until the email is sent. On the other hand, the server will try to send the email as soon as it's saved in the pickup directory so it's almost immediate (just without blocking the call).
You can use a background thread like described in other answers. You'll need to be careful with this option as the thread can end unexpectedly before it finishes its job. You'll need to add some code to make sure this works reliably (personally, I'd prefer not to use this option).
Using a messaging queue server like MSMQ. This is more work and you probably should only look into this if you have a large scale application or have good reasons not to use the first option with the pickup directory.
There are a few ways you could do this.
You could store enough details about the message in the database, and write a windows service to loop through them and send the email. When the user submits the form it just inserts the required data about the message and trusts the service will pick it up. Almost an email queue which you said you didn't want, but you're going to end up in a queue situation with almost any solution.
Another option would be to drop in NServiceBus. Use that for these kinds of tasks.
I typically compile the message body and store that in a table in the db along with the from and to addresses, a subject, and a timestamp indicating when the email was sent. Then I have a background task check the table periodically and pull any that haven't been sent. This task attempts to send each email and updates the timestamp accordingly. One advantage of storing the compiled message body up front is that the background task doesn't have to do any processing of context-specific data, and therefore can be pretty darn simple.
Whenever an operation like is hingent upon an event, there is always the possibility something will go wrong.
In ASP.NET you can spawn multiple threads and have those threads do the action. Make sure you tell the thread it's a background thread, otherwise ASP.NET might way for the thread to finish before rendering your page:
myThread.IsBackground = true;
I know you said you didn't want to use system process or scheduled tasks, but a windows service would be a viable approach to this as well. The approach would be to use MS Queue, or save the actions needing to be done in a DataBase table. Then have a windows service check every minute or so and do those actions.
This way, if something fails (Email server down) those emails / actions can still be done.
They will also be recorded for audit's (which is very nice to have).
This method allows you're web site to function as a website while offloading these tasks to another service. The last thing you need is for multiple ASP.NET processes to be used up waiting for emails to send. let something else handle that.