i implement a small bulk-mail sending tool in rails based on the Amazon SES service and action mailer. i read that amazon queues my sent messages before sending them out itself.
so my question: does that mean i don't need to implement a message-queue myself (e.g. 50 mails per 5 minutes) against blacklisting and does amazon that job for me and i just transfer 5000 mails to it?
You need to divide them into groups of 50 (see documentation Note at http://docs.amazonwebservices.com/ses/latest/DeveloperGuide/) first. Also see "Managing your Sending Activity" on that page (it's ajax-driven, so there's no other URL). I would use Delayed Job for the queue: http://railscasts.com/episodes/171-delayed-job.
Related
I'm in the process of planning the development of a mail-server to hand the sending of email across our multiple websites. Below is a description of what we are planning to implement and I'd like your opinion/suggestions.
We use ASP.NET MVC and have many web-sites hosted on Azure. We currently send mail internally within each of the web applications using SMTPServer.Send(). Obviously this is not the ideal way to send emails when you have a decently busy set of websites because the send mail call is blocking and cannot guarantee mails are sent. With this I'm worried out getting an influx of mail requests when we launch our next website (we think it'll get a decent amount of traffic and lots of emails will be sent).
My plan of action : to build a centralised mail-server that runs in the background (we use azure and this will be simply another web-application). When each one of our web applications wants to send a mail, instead of doing this internally, it'll call a web method on the mail-server called sendMail() this function will accept certain parameters and insert the mail parameters, content etc. into a database. The mail server will then poll the database at fixed time intervals, select a set of unsent emails and attempt to send them using the same SMTPServer.Send() function. If an email fails for some reason we won't flag it as sent and in the next poll interval the email will be selected again and another send attempt will be made. (we will cap the number of send attempts to say 20).
This will allow each of the websites to run smoothly without having loads of blocking send mail calls internally and the mail server will handle all the sending sequentially and in a controlled environment as a separate standalone web-application.
Thanks in advance!
Looks like a good design, Don't know the entire scenario which let to you building something like an email server. The problem has been solved well by using a service that already exist like Office 365.
Your design is good, My suggestions would be the following,
You can use Azure WebJobs to build the polling agent. You can make the web job run as a scheduled web job that does the polling and sending the mail and it can be written very clean as a simple console app.
You can use Azure API App to build the SendMail() call and you can use the Azure AD Auth on the API to authenticate the caller of the API using the Authentication and Authorization feature to easily secure your email server. You can also enable CORS easily as well to make sure you receive requests from other websites and process it.
Some issues I foresee for you,
Volume and Scaling : You can only process so much email between each polling. If you cannot then you will need to create another polling agent which will making things complicated as they need to know they are picking different sets of emails to send. If you volume is going to be low you should be fine.
Challenge : Why can't the websites send the mail them selves ? And then record it on the database for tracking. All you have to do build module or a component that they use on their web page to create and send the mail. Polymer 1.0 works well for this scenario.
Hope this helps to get you started.
Is it possible for me to get a message from the SQS queue based on the message ID with the Amazon PHP SDK? Do I have to just grab all of the messages on the queue and then filter it on my server?
My server receives a SNS instigated request with a queue message Id and I'm having to filter the message from an array of messages from SQS.
The purpose of a queue is to use it as a buffer to store messages before a certain processing task. This should not be confused with a storage service or a database service. Amazon SQS allows a process to retrieve messages from a queue (buffer) and process them as needed. If needed Standard or FIFO queues can be used.
In answer to your question: SQS does not provide a mechanism to retrieve by Message ID. So as you suggested, you can have separate workers to retrieve all messages in parallel and look for the message with the ID you want. (This can be exhaustive)
Since your use case is similar to that of a storage service, I suggest writing to a storage service and retrieving from it based on a column named "Message ID".
Just so that it may help someone else. I couldn't find a straight forward method to do this. Instead, implementing another set of queue workers to delegate the tasks solved the performance issue. These workers performed only two tasks
Retrieve queue item
Associative valid id
Send it to the processing server (after performing load checks, availability etc)
I would have to know more about your use case to know, but it sounds like you are using SQS as a database. Might I recommend instead of sending messages to SQS and sending the message ID to SNS, instead adding a row in DynamoDB, and sending to key through SNS?
Or even better yet, just send the raw data through SNS?
I have some questions about ActionMailer :
How does Actionmailer connect to a smtp server ?
Are the connections concurrent or parallel if the number of emails high > 1000 ?
How will sending out emails like facebook does ( 1000's in numbers ) as immediate emails affect the ruby on rails application and how would actionmailer handle it ?
Any other solution/plugin to send out large number emails from a RoR application apart ActionMailer?
------------------------------------------------added :
We need to send out at least 1000 emails per 15 minutes . We are using a Notes Domino server as our smtp server .! what is the possible architecture for this kind of problem. We are already storing the emails in the database to send them later , but what is needed is the sending approach !
The usual thing is to create a background job to send email. ActionMailer is very good for single emails but does tend to run into trouble after sending multiple emails as each one can take several seconds to complete. That's why I created PostageApp to help solve those problems.
Some services on the market to help you with sending lots of email from Rails:
MailGun
SendGrid
PostmarkApp
MailChimp
Mailjet
PostageApp
All of these have ways of sending multiple messages with a single API call or SMTP transaction.
1) Actionmailer connects to your smtp server via a set of parameters including a host, port and protocol.
3) The effect will be a slow site as a result of the many synchronous tasks being executed.
2 & 4)
Actionmailer is a bit too slow to be sending out a ton of emails under load, remember that it is a synchronous operation and as such its not really the sort of thing you want to be doing a lot on a busy site.
To be honest you're better off not sending that quantity of email from your website. It's not really designed to be used in such a way. If I had to send that sort of volume I'd look at doing the work in the background, something like Delayed Job would work well here or one of the many async rails mailers found here would do the trick.
What you really want to look at here is the requirement that you're trying to fulfil, is it absolutely necessary that the website be responsible for sending the mail in a synchronous fashion? In most cases the answer to that question is no. If you can, you'll be far better off deferring this sort of task to another part of your system, keep your site as lean and focused as you can.
Simple solution here for you...
Sidekiq or Resque
I'd highly recommend Sidekiq as it's not near as server intense for running multiple workers for this one - only be careful with concurrency issues (make sure you don't have 2 workers pick up the same job and send duplicate emails that is).
Say you set 20 Sidekiq workers, each should be able to send an email every 2-4 seconds, you're looking at an easy 300-600 per minute.
DO NOT try to do this without background workers like Sidekiq, Resque, or DelayedJob. You will freeze your entire app if you try sending in app with any large amount of emails. Even sending activation emails in app and what not will cause you unnecessary slow down issues.
I'd have one Worker that handles the queueing periodically and another Worker class that handles the sending. We're using Resque (6 workers maybe?) for this on an older app (pre-sidekiq) to send around 500 emails every 5 minutes with no issues.
You can aways use a third party like someone mentioned. Sendgrid is decent. But that wasn't the question, this is how you do it yourself simply and easily.
You define the SMTP settings in a config file if left blank it uses sendMail local
concurrent
multiple handlers
Is there a bulk email plugin for Rails apps?
you may also do 1000.times do email.deliver but it will probably collapse ur server
I've read most of the other answers on this topic, but a lot of them related to either third-party services like MailChimp (which I'm not necessarily opposed to) or how not to upset the host's email server.
I believe this case is unique so that it'll contribute...
I have my own DigitalOcean droplet running a rails app. I need to send out 100-1000 emails every so often, each with a unique message (a link I'm using for tracking clicks originating from the email).
I'm also operating my own iRedMail server.
Can someone recommend how to best-handle this task? I was going to simply cycle through the list of emails and use the template.html.erb to drop in my link, but what types of problems might I run into?
Thank you!
You should decouple your Rails App from the mail sending so that you don't have to wait in your view for the mails to be sent (assuming that you click on something that triggers the start of your mail sending). Use something like delayed_job or another queueing mechanism that Rails offers and only queue up the sending job of the e-mails. Then when the queue comes to execute the particular job you can customize the message with an HTML part and a text part or whatever else you need and pass them on individually to your MTA.
I try to create a mailing list feature in Rails which is based on delayed_jobs. For now I send mails by iterating over a users table and .deliver a mail to every mail address.
how can i integrate it into delayed_jobs, so it sends 50 mails every 5 minutes and remembers which adresses are already done? do i need to make a seperate table where i store all sent mails and check back everytime i send another 50 mails?
thanks in advance.
You will probably want to have table entries for sent emails. That way it serves as an audit trail if processes go down or somehow fail.
Suggest you look at doing this with an elastic cloud database like MongoLab, MongoHQ or SimpleDB. (Mongo-based services make it easy to extend the schema for new email entries.)
If you do that, then a cloud worker queue like SimpleWorker can make it easy to send out lots of emails concurrently or in batches to get around any rate limits. (full disclosure: I work at Iron.io/SimpleWorker)
You're taking a good approach to bundle multiple email sends into a single worker task to amortize the worker setup costs. With an elastic cloud worker system, you could have master workers come off schedule and then queue up a number of slave worker tasks, each with a set of users to send.
With table entries, you can then go back through the data tables and address any emails that failed or didn't go through.
50 emails is not really so much can be sent in seconds, I think. Use foreverb for sending emails every 5 minutes.
Let delayed job do all the work:
User.all.each_with_index |user, index|
Mailer.delay({:run_at => ((index / 50) * 5).minutes.from_now}).send_newsletter(user)
end
This should work but untested.