I have an application that needs to occasionally send an email blast to the entire user base when an admin does something. This was working fine, but when there are a lot of users, the page for the admin will wait until all the mail is sent, which is undesirable.
To mitigate this, I tried sending email in a new thread:
t = Thread.new do
User.all.each do |user|
Mailer.email(user).deliver
end
end
at_exit{ t.join }
This worked fine, but then in my test suite, I can't test to ensure the email sending works:
# This test now fails with the new Thread above
test "admin action should send email blast" do
assert_difference("ActionMailer::Base.deliveries.count", User.count) do
post :action
end
end
So my questions are:
Is this method the best way to send email in a new Thread? Or is there a gem available that handles this kind of interaction?
How can I test that the emails are sent in my test suite if the sending is done in a new thread? Is there a way to check to wait for all threads to finish?
In rails 4.2 there's a special class that handles jobs called ActiveJob, active job allows you to queue long tasks for another process to handle them in the background, also you can queue tasks for a certain time, like for example "send this email tomorrow at 8 am".
For these queues to get handled you need to choose a backend to handle them, here's a list of backends that support ActiveJob
Each has it's pros and cons, sidekiq for example is a multithread handler, so it uses lower memory, while for example sucker punch uses the same thread as the main server, so it uses a lot less memory, suitable if you have a low memory server that can't handle a second ruby thread.
As for the testing part, rails guide already explains how to test your emails and test things like if emails has been queued or not, and test that the right template was rendered, and if it contains the right text.
I think you want to send mail asynchronously, For that you may use many gems like - delayed jobs, sidekiq etc. I would personally recommend to use sidekiq as its faster and used Redis in memory db behind the scene.
With Rails 4.2, Active jobs are introduced, so using it has advantage that you can switch from one queuing system to other without having any worry at any time and you can specify which queing machinery yiu want like
module YourApp
class Application < Rails::Application
# Be sure to have the adapter's gem in your Gemfile
# and follow the adapter's specific installation
# and deployment instructions.
config.active_job.queue_adapter = :sidekiq
end
end
I prefer using a queue based approach for async tasks. [Delayed job] (https://github.com/collectiveidea/delayed_job) is one option, but I prefer using [sidekiq] (http://sidekiq.org).
[Here] (http://blog.remarkablelabs.com/2013/01/using-sidekiq-to-send-emails-asynchronously) is an example of sending emails asynchronously using sidekiq.
Best way to send emails in separate threads is to use delayed jobs or similar gem.
https://github.com/collectiveidea/delayed_job
With delayed jobs gem, you can also send emails or do something else in different threads at scheduled time.
Another option is to use sidekiq, install sidekiq gem and add this line to application.rb file
config.active_job.queue_adapter = :sidekiq
Related
i'm interested in running a rails app with sidekiq, and without active job. i find that if active job is there, people get confused if they should use active job apis or sidekiqs, and i want to use sidekiq's apis exclusively for performance reasons.
does current versions of sidekiq (i.e. version 7) support active mailer without active job's deliver_later? i think the historical way to do this was through "delayed extensions" but i see now this has been removed in sidekiq 7:
Delayed extensions provide a very easy and simple way to make method calls asynchronous. By default, all class methods and ActionMailer deliveries can be performed asynchronously.
They were disabled in Sidekiq 5 and removed in Sidekiq 7.
https://github.com/mperham/sidekiq/wiki/Delayed-Extensions
so given that, is there a recommended way to use action mailer with sidekiq and without active job these days?
I've used this instructions and sent "Welcome" mail to my signed up user. But this makes the user wait for 5-8 seconds because the server is trying to complete this mail thing.
I don't want the user to wait until the mail is sent but immediately see the "Mail has sent" message. So this brings me background jobs in Rails.
There are many options like delayed_job, Resque etc for background jobs in Rails. But to use these kind of solutions, as I understand:
1- Create a background job
2- Run this job
Let's say I used one of the background job solutions, so then I need something else to run also this job, like cron job...
I think just for sending sign-up and password reminder emails, another easier solution should be possible. I mean like another external service that 1- I'll create a template for each kind of mail I'll send, 2- I'll pass some arguments like receiver_email, template_id, receiver_username, password_link etc... With that way, I won't need any background job, and the user will not wait.
I came across some other gem called "sucher_punch" but as I understand from the people's messages and posted problems, with using heroku, this gem can fail for some reasons of heroku dynos and the mail may not be sent, and you don't know it.
Anyway, what is the general way that rails developers handle this email issue? Maybe I can also use sendgrid like the way I explained above, can I ?
Sending emails in the background is such a common use case, Rails 4.2 introduced a #deliver_later method in ActionMailer to provide seamless ActiveJob integration.
You don't need to set up a cron job to check if there are any jobs in the background queue. Sidekiq, Resque or DelayedJob will take care of that for you.
It seems Sendgrid does allow creating templates and sending variable content to fill them up, but that feature doesn't undermine the benefits of making that call asynchronously. In fact, deferring it to the background also has the added benefit of not disrupting user experience if the external resource(Sendgrid) is unavailable.
You should try installing one of the background processing solutions you mentioned(I recommend sidekiq) and take advantage of the ActionMailer + ActiveJob integration.
I have some questions about ActionMailer :
How does Actionmailer connect to a smtp server ?
Are the connections concurrent or parallel if the number of emails high > 1000 ?
How will sending out emails like facebook does ( 1000's in numbers ) as immediate emails affect the ruby on rails application and how would actionmailer handle it ?
Any other solution/plugin to send out large number emails from a RoR application apart ActionMailer?
------------------------------------------------added :
We need to send out at least 1000 emails per 15 minutes . We are using a Notes Domino server as our smtp server .! what is the possible architecture for this kind of problem. We are already storing the emails in the database to send them later , but what is needed is the sending approach !
The usual thing is to create a background job to send email. ActionMailer is very good for single emails but does tend to run into trouble after sending multiple emails as each one can take several seconds to complete. That's why I created PostageApp to help solve those problems.
Some services on the market to help you with sending lots of email from Rails:
MailGun
SendGrid
PostmarkApp
MailChimp
Mailjet
PostageApp
All of these have ways of sending multiple messages with a single API call or SMTP transaction.
1) Actionmailer connects to your smtp server via a set of parameters including a host, port and protocol.
3) The effect will be a slow site as a result of the many synchronous tasks being executed.
2 & 4)
Actionmailer is a bit too slow to be sending out a ton of emails under load, remember that it is a synchronous operation and as such its not really the sort of thing you want to be doing a lot on a busy site.
To be honest you're better off not sending that quantity of email from your website. It's not really designed to be used in such a way. If I had to send that sort of volume I'd look at doing the work in the background, something like Delayed Job would work well here or one of the many async rails mailers found here would do the trick.
What you really want to look at here is the requirement that you're trying to fulfil, is it absolutely necessary that the website be responsible for sending the mail in a synchronous fashion? In most cases the answer to that question is no. If you can, you'll be far better off deferring this sort of task to another part of your system, keep your site as lean and focused as you can.
Simple solution here for you...
Sidekiq or Resque
I'd highly recommend Sidekiq as it's not near as server intense for running multiple workers for this one - only be careful with concurrency issues (make sure you don't have 2 workers pick up the same job and send duplicate emails that is).
Say you set 20 Sidekiq workers, each should be able to send an email every 2-4 seconds, you're looking at an easy 300-600 per minute.
DO NOT try to do this without background workers like Sidekiq, Resque, or DelayedJob. You will freeze your entire app if you try sending in app with any large amount of emails. Even sending activation emails in app and what not will cause you unnecessary slow down issues.
I'd have one Worker that handles the queueing periodically and another Worker class that handles the sending. We're using Resque (6 workers maybe?) for this on an older app (pre-sidekiq) to send around 500 emails every 5 minutes with no issues.
You can aways use a third party like someone mentioned. Sendgrid is decent. But that wasn't the question, this is how you do it yourself simply and easily.
You define the SMTP settings in a config file if left blank it uses sendMail local
concurrent
multiple handlers
Is there a bulk email plugin for Rails apps?
you may also do 1000.times do email.deliver but it will probably collapse ur server
I'm looking for a bit of guidance on writing acceptance tests for a process that uses Ryan Bates' private_pub gem (https://github.com/ryanb/private_pub) and Resque.
In my controller I have an execute action that queues up a file generation process in Resque and then makes an AJAX request to display a "waiting" modal window. The modal window contains the javascript to subscribe to a private_pub channel and wait for a message on that channel. Upon completion of the file generation process in the Resque worker, a message with the newly generated file name is pushed to private_pub. The javascript in the modal receives the message and uses the filename to kick off a download of the file in the browser.
I'm looking for suggestions on techniques to run tests on this process. I've considered mocking the private_pub server (not sure how to even accomplish this after much Googling) so that it simply returns a pre-staged file. Basically I think I need a way to exercise my workflow without the need to run a private_pub server and the whole Resque infrastructure in my test environment.
I'd recommend stubbing out the call to the Resque worker and simply returning what you'd expect it to return. Then test the Resque worker code in a separate test.
If your goal is specifically to test the functionality of the modal window, you could write a test that simply makes the call to the Resque worker and use Capybara to determine if the modal shows up. In this example, you would not want to stub the Resque call - simply end the test after you've found the modal window appears.
I need to setup a Ruby / Rails application which can accept incoming emails, the body of the emails will be ignored, each email will have one or more large attachments (graphics) up to 20MB in size, the app will receive approx 10-20 of these emails an hour.
I need to use a custom email address / domain.
Quick processing of the emails is not a big priority, however each email should get handled in at most 5 minutes after it arrives.
I was thinking of having all emails automatically forwarded from my info#mydomain.com mail account to a GMail inbox, then have a Rails lib script use MMS2R & TMail access GMail, download any new mails & add them to a message queue (delayed job), this script would be invoked by a cron job once a minute.
This solution just does not "smell right" & I would be concerned about its ability to scale; I don't want to load up the whole Rails stack each time the cron / script is run.
Does anyone have experience with incoming mail & attachment processing in Rails?
If your main concern is about loading rails environment every minute to run the script, I would suggest mailgun along with some cron alternatives.
I use a combination of bluepill + clockwork + delayed_job to handle this. I have custom jobs, enqueing which is as simple as Delayed::Job.enqueue MyJob.new. The logic is captured in job's perform method, which helps minimize dependencies if there are any.
clockwork is used to enqueue delayed jobs only loading minimal dependencies, keeping the memory footprint minimal.
bluepill is used to monitor the delayed workers and clockwork process and start them back up if they fail or restart if they are getting out of control.
General memory usage: 20MB(bluepill) + 25MB(clockwork) + 80-90MB(delayed worker)
Example of clockwork.rb:
ENV['RAILS_ENV'] ||= "development"
ENV['RACK_ENV'] = ENV['RAILS_ENV']
require 'clockwork'
require 'mongoid'
require 'delayed_job'
require 'delayed_job_mongoid'
include Clockwork
base_directory = File.absolute_path(File.dirname(__FILE__))
require File.join(base_directory, "../jobs/fetch_comments_job.rb")
require File.join(base_directory, "../jobs/news_letter_job.rb")
Mongoid.load!(File.join(base_directory, "../config/mongoid.yml"))
every(1.minute, "pop.comments") { Delayed::Job.enqueue FetchCommentsJob.new}
every(1.day, "send.newsletter", :at => '22:30') { Delayed::Job.enqueue NewsLetterJob.new}
I wrote a blog post about a number of possible options for receiving email in Rails. The cron approach works but but there can be a number of issues if the processing takes longer than your suggested 5 minutes and your then spinning up another instance whilst the old one is still going.
The alternatives look at how you can send the email directly to your web app so that you can use the existing app rather than having to spin up another instance each time the cron task runs. There are obviously potential downsides to doing this too but systems like CloudMailin allow you to send the attachments directly to S3 to get around this. You can then just use a background processing system to download the attachments and resize them.
I also had to modify paperclip a little to get it to resize the attachments.