I have an SNS topic called onboard-user. A lot of microservices subscribe to this topic
for ex:
Emai microservice - to send the email
System preferences microservice -To set the defaults
There is a queue in these microservices that subscribes to the onboard-user SNS topic.
Now, if the email to the user needs to be resent, how should it be handled? onboard-user should not get a new message in that case as that has its own purpose and we would not want all of the microservices listening to it to be notified.
Having an alternate SNS topic for send emails doesn't sound like such a good idea and an overkill. what's the right way to handle it?
Why would an email need to be resent? Due to a transient error or due to some business logic initiated by an end user? The solution depends on the problem. I would argue that if you detect the case where a message needs to be recent that you either: (A) maintain separate resend email SQS queue OR (B) put a new message on the existing email queue with a field indicating that this is a resend.
Since SQS is pretty cheap, I could lean towards (A).
If my understanding is correct, you have single SNS topic to do multiple processing and that's why multiple microservices are listening to it. Please correct, if that is incorrect.
If that is the case, the better option is to:
a. Segregate email processing - keep a separate SNS topic for email purpose e.g. 'inform-user' and use it for retrying
b. If you can't segregate, then the only option in above scenario is to have "state" in the message itself. That means having an attribute in the message to indicate that it is for "retrying" and then the listener microservice should process accordingly.
Related
It is possible to get notify when another client was connected to same topic which is subscribed by same topic in mqtt.
ex: a client subscribed to : app/id another client also subscribed to app/id then both clients will get a message about number of clients subscribed to this particular topic.
if it is possible then please let me know.
The whole point of a pub/sub architecture is to unlink the producer of information (publisher) from the consumer (subscriber). Producers just publish to a topic and the broker deals with routing that message to any consumers that may have a matching topic pattern subscription.
There is no way to count how many clients may receive a given message at any particular time for the following reasons:
A subscriber may be using a wildcard topic pattern that happens to match the topic of the published message.
A subscriber may have a persistent session with a matching topic pattern but if currently off line and will receive the queued message when it next connects
With shared subscriptions there could be any number of clients pooled together subscribed to a given topic pattern, but only one of them will receive any given message as they are load balanced round the group.
There is no guarantee that there are ever any subscribers, in the same way there is no end to end delivery guarantee only between a single client and the broker.
In a sensibly implemented broker, subscription topic pattern matches are only evaluated when a new message is published, so the best you could ever get is how many clients the last message with a given topic was delivered to, as there is no efficient way to calculate the number in advance.
In the same way as your last question about topics, if you think knowing this is required then you need to rethink your design or your use of MQTT.
I went through the AWS pinpoint documentation for checking delivery status of sent SMS message. But, I couldn't find any batter way than streaming SMS Event to either Amazon Kinesis Data Streams or Amazon Kinesis Data Firehose then sending data to Amazon S3 bucket or Amazon Redshift database and only after that to the application.
Here, my requirement is to send just verification code and get the delivery status of SMS message. Is there any better way of achieving this? Because the way I mentioned above is too lengthy and costly. Any help will be appreciated.
sendMessages() API returns status of SMS. It is assumed that this API may not instantly return send status as SMS may not get delivered instantly.
But since "callback", it takes time to return, works Asynchronously.
What is the status it is returning?
refer to: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Pinpoint.html#sendMessages-property
In callback, json "data" returned has "MessageResponse" has field "Result" has field "DeliveryStatus" which has possible values as below:
"SUCCESSFUL"
"THROTTLED"
"TEMPORARY_FAILURE"
"PERMANENT_FAILURE"
"UNKNOWN_FAILURE"
"OPT_OUT"
"DUPLICATE"
Why is status update through Kinesis, then S3 buckets, then Analytics is needed? Actually not needed. With Kinesis you get more detailed information.
On hard pressing simpler ways, AWS expert mentioned use Lamda service to directly send to some "Queue" (possibly another AMZ product) or call HTTP end point through Lamda service. All these services are costly and cost other resources too. "Total Cost of Ownership" just grows or these guys are "up-selling" more products.
I want to send transactional SMS. That's it. I need to know if SMS was delivered or not. That's it. But you end spending for Kinesis (how ever small or large cost), for S3, for Analytics Services, associated EC2, EBS everything.
I discussed hard with AMZ Architect, who settled to, with Kinesis you can directly poll and get status if you configure Pinpoint with Kinesis. So I get SMS event updates ALSO through Kinesis, which are "similar" to what you get by calling getMessages() API.
In SMS event stream directly through Kinesis, you get
"event_type" value as
"_SMS.BUFFERED"
"_SMS.SUCCESS"
"_SMS.FAILURE"
"_SMS.OPTOUT"
reference:
https://docs.aws.amazon.com/pinpoint/latest/developerguide/event-streams-data-sms.html
And value of "record_status" field you get
"SUCCESSFUL"
"DELIVERED"
"PENDING"
"INVALID"
"UNREACHABLE"
"UNKNOWN"
"BLOCKED"
"CARRIER_UNREACHABLE"
"SPAM"
"INVALID_MESSAGE"
"CARRIER_BLOCKED"
"TTL_EXPIRED"
"MAX_PRICE_EXCEEDED"
Some of these status code may be irrelevant to your case. e.g. "TTL_EXPIRED" is possible value if you are sending "delayed" SMS.
Kinesis, delivered more detailed, but may not be practically useful.
If you exceed your max SMS cost Quota with Kinesis, you get the reason. With sendMessages() API you won't. But you can check SMS quota through APIs.
refer to: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ServiceQuotas.html#getServiceQuota-property
I strongly believe, if sending "Transactional" SMS which are by nature priority delivered, you don't need Kinesis.
Amazon is overselling by entangling services into composite solutions, in some cases. Anyways, they are not wrong, Pinpoint is "Marketing" product to do Campaigns, User Journey and Conversion Analysis. Which is bigger business case for Amazon that "just sending SMS". You may not get best advise for your Usecase, so negotiate hard to save your $s.
I made an instant messaging app using MQTT protocol.
I want to add some extra data about messages in payload like sent time ( server time not client time ) and also provide kind of server side payload sanitizing.
Is it a good idea to add a third party client with superuser privileges between message sender and message receiver on broker's local machine to do this job ?
or is there any better idea ?
by the way I'm using EMQTT as message broker.
From a pure security view having direct peer to peer traffic (without filtering and sanitising) sounds like a dangerous idea. (At least in the Internet-of-things domain I would clearly object against it.)
Why? Because the clients are outside of your control (i.e. a hacker can re-engineer) and inject any traffic to exploit security holes on the receiving side of other clients.
So sanitising on the server side sounds like a very good idea.
I would suggest two topics: One (inbound) topic the clients use to publish messages, and another (outbound) topic used by clients to subscribe to messages. A server side component would then read the messages from inbound topic, sanitize it and publish to the outbound topic.
This de-coupeling makes it also easier to introduce MQTT payload changed: If you update the payload in a non-compatible way, introduce a new inbound topic and keep the old inbound topic too. This allows you to support old and new clients during the transition phase.
Is it possible for me to get a message from the SQS queue based on the message ID with the Amazon PHP SDK? Do I have to just grab all of the messages on the queue and then filter it on my server?
My server receives a SNS instigated request with a queue message Id and I'm having to filter the message from an array of messages from SQS.
The purpose of a queue is to use it as a buffer to store messages before a certain processing task. This should not be confused with a storage service or a database service. Amazon SQS allows a process to retrieve messages from a queue (buffer) and process them as needed. If needed Standard or FIFO queues can be used.
In answer to your question: SQS does not provide a mechanism to retrieve by Message ID. So as you suggested, you can have separate workers to retrieve all messages in parallel and look for the message with the ID you want. (This can be exhaustive)
Since your use case is similar to that of a storage service, I suggest writing to a storage service and retrieving from it based on a column named "Message ID".
Just so that it may help someone else. I couldn't find a straight forward method to do this. Instead, implementing another set of queue workers to delegate the tasks solved the performance issue. These workers performed only two tasks
Retrieve queue item
Associative valid id
Send it to the processing server (after performing load checks, availability etc)
I would have to know more about your use case to know, but it sounds like you are using SQS as a database. Might I recommend instead of sending messages to SQS and sending the message ID to SNS, instead adding a row in DynamoDB, and sending to key through SNS?
Or even better yet, just send the raw data through SNS?
I'm building a web application were users need to ability to send out mass emails. The application is a ticket site where individuals sell tickets to various events. In turn, events have promoters that sell tickets. The sponsor of the event needs to ability to send mass emails to their promoters. An event may have hundreds of promoters. So I'm assuming looping through each promoter and sending an email wont cut it.
Does rails 3.0 provide some kind of mass email functionality?
Thanks,
Brian
I don't think you should have a problem looping through all the promoters and sending each one an email. However, you would want to do that as a background process.
Check out delayed-job or http://www.simpleworker.com/. Both of these will let you load the email into a background job, and allow you to redirect the user in a timely manner while the emails get sent in the background.
Solomon is right, you'll definitely want to do this in a background task, which is pretty simple with something like delayed job. Have you thought about how the messages will actually be delivered though? What mail server you are going to use? Many providers won't let you send out hundreds of messages at a time, and you are likely to run into spam issues if you try to send that volume of messages from a personal email account. You may want to take a look at a service like Mailgun that specializes in this. From their FAQ:
Why not just use Sendmail + Postfix + Courier IMAP?
You can but you should be aware that there is a constant battle raging
between good and evil (i.e., spam) in the email universe. In order to
be on the 'good' side of that battle and get your email delivered
there are numerous things you need to do. You need to have the right
infrastructure and register your IP and Domain appropriately. Also,
you need to have a history of email sending that complies with ESPs
rules in order to build a good reputation.
Moreover, if you are going to receive, store and host emails, you
better be prepared for maintaining this orchestra of software, take
care of backups, hardware failures, security patches and monitoring.
Stop kidding yourself, it's not 1998 anymore.
I'm not affiliated with them in any way, but in my experience getting rails to send email is trivial compared to dealing with getting it delivered successfully by the mail server.