I try to create a mailing list feature in Rails which is based on delayed_jobs. For now I send mails by iterating over a users table and .deliver a mail to every mail address.
how can i integrate it into delayed_jobs, so it sends 50 mails every 5 minutes and remembers which adresses are already done? do i need to make a seperate table where i store all sent mails and check back everytime i send another 50 mails?
thanks in advance.
You will probably want to have table entries for sent emails. That way it serves as an audit trail if processes go down or somehow fail.
Suggest you look at doing this with an elastic cloud database like MongoLab, MongoHQ or SimpleDB. (Mongo-based services make it easy to extend the schema for new email entries.)
If you do that, then a cloud worker queue like SimpleWorker can make it easy to send out lots of emails concurrently or in batches to get around any rate limits. (full disclosure: I work at Iron.io/SimpleWorker)
You're taking a good approach to bundle multiple email sends into a single worker task to amortize the worker setup costs. With an elastic cloud worker system, you could have master workers come off schedule and then queue up a number of slave worker tasks, each with a set of users to send.
With table entries, you can then go back through the data tables and address any emails that failed or didn't go through.
50 emails is not really so much can be sent in seconds, I think. Use foreverb for sending emails every 5 minutes.
Let delayed job do all the work:
User.all.each_with_index |user, index|
Mailer.delay({:run_at => ((index / 50) * 5).minutes.from_now}).send_newsletter(user)
end
This should work but untested.
Related
We use Microsoft Graph to subscribe to webhooks from emails. Additionally, as a backup procedure we also crawl the messages directly.
We crawl around 5 million emails a day, and we see that each day consistently around 1%-2% of these emails are not sent to us via the webhook, although the subscription for this principal is active (and other email notifications from this user are indeed sent).
Is there any logic on the Microsoft Graph side to not send webhooks for certain types of emails by design? or is it just a problem on the notification mechanism?
(Obviously crawling them, as we do now, is a viable backup option, but that also means the processing of the email will be delayed)
I currently have a similar webhook setup and we get around 200-300 emails and I notice that the subscription service usually misses out on 1-2 per day since sometimes some emails come around at the same time. I have also noticed that the data structure is an array of objects when we get two or more emails at the same time. What we have put into place is basically a cron scheduled script that checks the mailbox on specific time intervals, such as every 5 minutes, every 10 minutes and so on. This is the only solution that has worked for my application to capture every single email.
I have some questions about ActionMailer :
How does Actionmailer connect to a smtp server ?
Are the connections concurrent or parallel if the number of emails high > 1000 ?
How will sending out emails like facebook does ( 1000's in numbers ) as immediate emails affect the ruby on rails application and how would actionmailer handle it ?
Any other solution/plugin to send out large number emails from a RoR application apart ActionMailer?
------------------------------------------------added :
We need to send out at least 1000 emails per 15 minutes . We are using a Notes Domino server as our smtp server .! what is the possible architecture for this kind of problem. We are already storing the emails in the database to send them later , but what is needed is the sending approach !
The usual thing is to create a background job to send email. ActionMailer is very good for single emails but does tend to run into trouble after sending multiple emails as each one can take several seconds to complete. That's why I created PostageApp to help solve those problems.
Some services on the market to help you with sending lots of email from Rails:
MailGun
SendGrid
PostmarkApp
MailChimp
Mailjet
PostageApp
All of these have ways of sending multiple messages with a single API call or SMTP transaction.
1) Actionmailer connects to your smtp server via a set of parameters including a host, port and protocol.
3) The effect will be a slow site as a result of the many synchronous tasks being executed.
2 & 4)
Actionmailer is a bit too slow to be sending out a ton of emails under load, remember that it is a synchronous operation and as such its not really the sort of thing you want to be doing a lot on a busy site.
To be honest you're better off not sending that quantity of email from your website. It's not really designed to be used in such a way. If I had to send that sort of volume I'd look at doing the work in the background, something like Delayed Job would work well here or one of the many async rails mailers found here would do the trick.
What you really want to look at here is the requirement that you're trying to fulfil, is it absolutely necessary that the website be responsible for sending the mail in a synchronous fashion? In most cases the answer to that question is no. If you can, you'll be far better off deferring this sort of task to another part of your system, keep your site as lean and focused as you can.
Simple solution here for you...
Sidekiq or Resque
I'd highly recommend Sidekiq as it's not near as server intense for running multiple workers for this one - only be careful with concurrency issues (make sure you don't have 2 workers pick up the same job and send duplicate emails that is).
Say you set 20 Sidekiq workers, each should be able to send an email every 2-4 seconds, you're looking at an easy 300-600 per minute.
DO NOT try to do this without background workers like Sidekiq, Resque, or DelayedJob. You will freeze your entire app if you try sending in app with any large amount of emails. Even sending activation emails in app and what not will cause you unnecessary slow down issues.
I'd have one Worker that handles the queueing periodically and another Worker class that handles the sending. We're using Resque (6 workers maybe?) for this on an older app (pre-sidekiq) to send around 500 emails every 5 minutes with no issues.
You can aways use a third party like someone mentioned. Sendgrid is decent. But that wasn't the question, this is how you do it yourself simply and easily.
You define the SMTP settings in a config file if left blank it uses sendMail local
concurrent
multiple handlers
Is there a bulk email plugin for Rails apps?
you may also do 1000.times do email.deliver but it will probably collapse ur server
Sorry for the basic question about Sidekiq's delaying ActionMailer. As per this article, Sidekiq can delay sending out emails by just saying UserMailer.delay_for(1.hour).....
Does this mean this is handled in the background now, or does it mean that it simply just delays sending the email out for an hour but once that hour comes, then the email is basically being sent like a regular ActionMailer, which slows down response time?
Or is it that if I truly want to do this in the background then I would have to do the other sidekiq stuff like putting it in a specific Worker and then firing it up that way?
Also, separately, if I do just do it via UserMailer.delay..., I presume I won't need a worker dyno on Heroku to save some money, correct?
Thanks for the help!
Yes, for emails you don't need to do anything else. It's like calling the Mailer 1 hour later. You just need to make sure you don't pass any complex objects into the mailer, for example a user object, you should only pass the user_id, because it will be stored in redis. On the Mailer fetch the user object with the given id.
i implement a small bulk-mail sending tool in rails based on the Amazon SES service and action mailer. i read that amazon queues my sent messages before sending them out itself.
so my question: does that mean i don't need to implement a message-queue myself (e.g. 50 mails per 5 minutes) against blacklisting and does amazon that job for me and i just transfer 5000 mails to it?
You need to divide them into groups of 50 (see documentation Note at http://docs.amazonwebservices.com/ses/latest/DeveloperGuide/) first. Also see "Managing your Sending Activity" on that page (it's ajax-driven, so there's no other URL). I would use Delayed Job for the queue: http://railscasts.com/episodes/171-delayed-job.
In my rails app, I'm using the SendGrid parse API which posts mail to my server. Every now and then SendGrid's Parse API submits the same email twice.
When I get a posted mail I place it in the IncomingMail model. so in order to prevent this double submitting issue, I look at each IncomingMail when processing to see if there is a duplicate in the table within the last minute. That tested great on development, it caught all the double submissions.
Now I pushed that live to heroku, where I have 2+ dynos and it didn't work. My guess being that it has something to do with replication. So that being the case, how can scalable sites with multiple server deal with something like this?
Thanks
You should look at using a background job queue. Heroku has "Workers" (which was Delayed Job). Rather than sending the email immediately, you push it onto the queue. Then one or more Heroku 'workers' need to be added to your account, and each one will pull jobs in sequence. This means there can be a short delay (depending on load) before the email is sent, but this delay is not presented to the user, and should there be a lot of email to send you just add more workers.
Waiting for an external service like an email provider on each user action is dangerous because any network problem will take down your site as several users have to 'wait' for their HTTP requests to be responded to while Heroku is blocked with these third party calls.
In this situation with workers each job would fail but would be retried and eventually succeed.
This sounds like it could be a transaction issue. If you have multiple workers running simultaneously their operation may be 'interleaved'. For instance this sequence of events would result in 2 mails being sent.
Worker A : Checks for an existing record and doesn't find one
Worker B : Checks for an existing record and doesn't find one
Worker A : Post to Sendgrid
Worker B : Post to Sendgrid
You could wrap everything in a transaction to keep this from happening. Something like this should do it.
class IncomingMail < ActiveRecord::Base
def check_and_send(email_address)
transaction do
# your existing code for preventing duplicates and sending
end
end
end