I have this rails app that's running on our local intranet, with a thousand regular users. I am looking to integrate it with our email server(MS Exchange). Basically -
1) For each user, the app should fetch any new messages in their inbox from the mail-server, parse it, and file it in the database.
I could implement it with ruby/net-imap, but I am worried about performance. As I have mentioned there are a thousand users, each with a different email address. Polling the email server every x minutes wouldn for a thousand users could drastically reduce performance.
It looks like IMAP IDLE with ruby/net-imap could be an option, the mail server pushing a notification to my app whenever there's a new mail(IPhone/Outlook style). But it seems, for that i'll have to keep an imap connection open with the server for that. So if there are 100 users logged in at a time, that would mean 100 connections to mail-server. Will that drastically slow the app?
Or does anyone know of any other options (Btw, I don't have control over the mail-server)
We are using MRI, but could easily switch to jruby too.
This might help... I know it is for gmail or not exactly what you're looking for, but you might be able to adapt it to your problem.
There are also some libraries to handle imap with ruby such as Chilkat Ruby IMAP Library 2.1, but I haven't tried any yet.
Related
I have problems sending mail from a ruby production app on heroku by a gmail smtp account.
Here i found the solution
When i apply the solution all works wel, but after a while the problems comes back.
Do i have to set something in my google account to make it permanent ?
I think it has to do with the fact that i am using a free heroku account.
The servers stops after a while an perhaps has to login in gmail when it starts up again.
There's no way to make it permanent because Gmail doesn't want its service to be used to send out automated e-mails(*).
There are some providers that will make it much easier for you:
https://sendgrid.com/pricing/ has a free plan (up to 100 emails/day)
https://postmarkapp.com/pricing gives you 10k emails for 10$)
or take your pick, because the market seem to be pretty saturated: https://duckduckgo.com/?q=e-mail+delivery+service&ia=web
(*) What I suspect they do (though I could not find info about it) is that they can see if "same e-mails" are sent within short intervals.
You might try your luck (if it makes sens, when for example it's a toy project) to alter each e-mail contents to make it seem like different e-mail each time. But it will be just easier to go with free option in any other providers that actually want you to use it in your app.
I have some questions about ActionMailer :
How does Actionmailer connect to a smtp server ?
Are the connections concurrent or parallel if the number of emails high > 1000 ?
How will sending out emails like facebook does ( 1000's in numbers ) as immediate emails affect the ruby on rails application and how would actionmailer handle it ?
Any other solution/plugin to send out large number emails from a RoR application apart ActionMailer?
------------------------------------------------added :
We need to send out at least 1000 emails per 15 minutes . We are using a Notes Domino server as our smtp server .! what is the possible architecture for this kind of problem. We are already storing the emails in the database to send them later , but what is needed is the sending approach !
The usual thing is to create a background job to send email. ActionMailer is very good for single emails but does tend to run into trouble after sending multiple emails as each one can take several seconds to complete. That's why I created PostageApp to help solve those problems.
Some services on the market to help you with sending lots of email from Rails:
MailGun
SendGrid
PostmarkApp
MailChimp
Mailjet
PostageApp
All of these have ways of sending multiple messages with a single API call or SMTP transaction.
1) Actionmailer connects to your smtp server via a set of parameters including a host, port and protocol.
3) The effect will be a slow site as a result of the many synchronous tasks being executed.
2 & 4)
Actionmailer is a bit too slow to be sending out a ton of emails under load, remember that it is a synchronous operation and as such its not really the sort of thing you want to be doing a lot on a busy site.
To be honest you're better off not sending that quantity of email from your website. It's not really designed to be used in such a way. If I had to send that sort of volume I'd look at doing the work in the background, something like Delayed Job would work well here or one of the many async rails mailers found here would do the trick.
What you really want to look at here is the requirement that you're trying to fulfil, is it absolutely necessary that the website be responsible for sending the mail in a synchronous fashion? In most cases the answer to that question is no. If you can, you'll be far better off deferring this sort of task to another part of your system, keep your site as lean and focused as you can.
Simple solution here for you...
Sidekiq or Resque
I'd highly recommend Sidekiq as it's not near as server intense for running multiple workers for this one - only be careful with concurrency issues (make sure you don't have 2 workers pick up the same job and send duplicate emails that is).
Say you set 20 Sidekiq workers, each should be able to send an email every 2-4 seconds, you're looking at an easy 300-600 per minute.
DO NOT try to do this without background workers like Sidekiq, Resque, or DelayedJob. You will freeze your entire app if you try sending in app with any large amount of emails. Even sending activation emails in app and what not will cause you unnecessary slow down issues.
I'd have one Worker that handles the queueing periodically and another Worker class that handles the sending. We're using Resque (6 workers maybe?) for this on an older app (pre-sidekiq) to send around 500 emails every 5 minutes with no issues.
You can aways use a third party like someone mentioned. Sendgrid is decent. But that wasn't the question, this is how you do it yourself simply and easily.
You define the SMTP settings in a config file if left blank it uses sendMail local
concurrent
multiple handlers
Is there a bulk email plugin for Rails apps?
you may also do 1000.times do email.deliver but it will probably collapse ur server
I am currently using a Mac Mini Server running OS X Server. The server hosts several Ruby on Rails applications using Phusion Passenger where two of them uses a PostgreSQL database. One of these applications has a mailing list database that uses ActionMailer to send emails. As of the writing of this post the application will send 224 emails when we send a newsletter out.
We were using a GoDaddy email address in the SMTP configuration. The emails took only five minutes to send. However GoDaddy only allows 250 emails to be sent per day. They also only allow you to purchase relays 50 at a time with a three month minimum. We were not sure how well that would work for us with the growth of our mailing list. At the rate we are growing we could add 50 people to the mailing list sooner than three months we would have already paid for. I felt that we should visit turning on the Mail Server on our Mac Mini Server.
After getting some questions answered from Apple I was able to get the Mail Server working where I can send and receive emails. However when I send those same emails it takes 30 minutes instead of five. The documentation for OS X Server still has old information from Lion Server referencing Server Admin Tools which no longer apply for OS X Server.
I was hoping that the time to send emails might be the same since the database for the Rails application and the mail server were on the same machine. However in general I understand in some cases there could be some contention with using the same resources. Is there is any way that I can speed up the sending of the emails? Unfortunately Server Admin Tools has not been provided for OS X Server. The documentation for OS X Server is for the most part worthless since they reference Server Admin Tools which no longer exists.
I have posted this on Server Fault and Ask Differently. However here I wanted to ask if there are specific things I can do in my Rails Application that may make the sending of emails using ActionMailer more efficient.
Any help would be appreciated.
We have a similar situation where we send anywhere from 100-300 emails out to job applicants based upon interview results. Sending email through ActionMailer is slow as it usually takes several seconds per email.
That said, sending emails is one of the classic use cases for a job queue. A similar question provided some helpful answers but I can tell you that I use Resque and ResqueMailer and have been very pleased with the results. It does require standing up a Redis service, but it is relatively painless to do so.
I have stopped using the Mail Service on my Mac Mini Server. I could not find a solution through Apple Technical Support. I suffered through this until a month ago when it took almost an hour to send our emails. I decided to look for other SMTP options.
I found out about Mandrill (created by the founders of MailChimp). They allow you to send 12,000/month for free then a minimal amount per 1000 emails (I think). All I had to do was to change my SMTP configuration to use Mandrill and add a couple of TXT DNS records and I was set.
Before I attempted the Mail Service I was using GoDaddy which took about eight minutes to send about 250 emails. I used Mandrill for the first time to send emails to my entire mailing list (almost 350) and it took less than three minutes. No ads or anything like that. I successfully send both GoDaddy and Mandrill emails. I have changed all my Rails applications to use the Mandrill SMTP settings. The reports available are wonderful.
Mandrill Web Site
How would i go about to receive mails in a Ruby on Rails application without going through a mail server like PostFix or to fetch them by pop3 etc.
What i was to do is to catch all mails sent to #mydomain.com and just do something with them in my application. I don't need to store the mails or anything like that.
Is this posible?
I just implemented this for my SAAS to autoprocess mailer-bounce notification messages.
Call me, call you?
You call me
You can set up a local mail server. It would then respond to an incoming email, and start up a rails executable to process the email. This method is NOT recommended since starting up rails is a big task (takes multiple secs and lots of memory). You don't want a Rails bad boy started up just because you received an email. You'd be writing your own DDOS attack. (Attacking yourself.)
I call you
Instead, poll for email on your own schedule by using a single job to process all currently waiting emails. You need to set up a background job handler since stock rails is focused on responding to web requests. I use delayed_job, but there are other alternatives including kicking off a cron job every so in often.
Another benefit is that you don't need to manage a mail server. Leave that headache to someone else. Then use the Ruby library net::imap to read the incoming mail and process it.
If your process doesn't recognize the email format, then forward the msg to a human for processing.
And be sure that if the process sends mail in addition to reading/processing it, that the process uses a different email address as its From address. Otherwise, odds are good that sometime along the way, you'll end up in an email loop and many gigabytes of messages going back and forth. For example, your process receives a message, responds to it, but in the meantime the sender (a human) has switched on vacation response. And your robot then responds to the vacation response..... oops....
Writing your own mail server
Re:
How would i go about to receive mails in a Ruby on Rails application without going through a mail server like PostFix or to fetch them by pop3 etc.
What i was to do is to catch all mails sent to #mydomain.com and just do something with them in my application. I don't need to store the mails or anything like that.
Direct answer: Yes, you could do this by writing an smtp server and setting up dns so your machine will be the mail destination for the domain. Your smtp server would process the messages on the fly, they would not be stored on your system at any point.
Is this a good idea? No, not at all. While appearances may be to the contrary, email is a store and forward system. Trying to avoid storing the messages before your app processes them is not smart. It would be a very very poor "optimization." However, using an access protocol (POP3 or IMAP) is a good way to avoid the costs of installing, configuring and managing a mail server.
You can do this if you write your own mail server, or if your mail server supports hooks to run external programs upon receipt of mail (e.g. procmail).
If you don't have procmail available (or, if on something like Exchange Server, don't feel like writing custom rules or extensions), you're simply better off using a pop3 library to fetch mail.
Obviously, writing a mail server is more difficult than any of the alternatives.
If you're mostly worried about checking potentially hundreds of email accounts, that's solvable by configuring your email server properly. If you're on a hosted provider, ask your server administrator about creating a "catch-all" account that routes all mail to unknown addresses to a single account.
If you're aiming to avoid having to poll a server, consider the IMAP IDLE command. I've successfully written a Ruby client that opens a connection to an IMAP server, and gets told by the server when new mail arrives.
When a user completes an order at my online store, he gets an email confirmation.
Currently we're sending this email via Gmail (which we chose over sendmail for greater portability) after we authorize the user's credit card and before we show him a confirmation message (i.e., synchronously).
It's working fine in development, but I'm wondering if this will cause a problem in production. Will it require making the user wait too long? Will many simultaneous Gmail connections get us in trouble? Any other general caveats?
If sending the emails synchronously will be a problem, could someone recommend an asynchronous solution (is ar_mailer any good?)
The main issue I can think of is that Gmail limits the amount of email you can send daily, so if you get too many orders a day it might break.
As they say :
"In an effort to fight spam and
prevent abuse, Google will temporarily
disable your account if you send a
message to more than 500 recipients or
if you send a large number of
undeliverable messages. If you use a
POP or IMAP client (Microsoft Outlook
or Apple Mail, e.g.), you may only
send a message to 100 people at a
time. Your account should be
re-enabled within 24 hours. "
http://mail.google.com/support/bin/answer.py?hl=en&answer=22839
I would recommend using sendmail on your server in order to have greater control over what's going on and don't depend on another service, especially when sendmail is not really complicated to set up.
The internet is not as resilient as some people would have you believe, the link between you and GMail will break at some point or GMail will go offline causing the user to think that they have not paid sucessfully.
I would put some other queue in place, sendmail sounds acceptable and you can't create your site now for where it 'might' be hosted in the future.
Ryan
If the server waits for the email to be sent before giving the user any feedback, were there problems connecting to the mailserver (timeouts, server down etc) the user request would timeout too and he wouldn't be told anything about the status of his order, so I believe you should really do this asynchronously.
Also, you should check whether doing that is even allowed by GMail's TOS. If that's not the case, you may check if that's allowed if you purchase one of their subscriptions. Also, there's surely a limit to the number of outgoing emails you may send within a given timeframe so if you're expecting your online store to be successful, you may hit that limit and bump into some nasty issue. If you're not self-hosting the site, you should check whether your host offers email servers (several plans include them for free) as then using your host's ISP would be the most obvious choice.
FACT: Gmail crashes. Not often, but it happens, and you can't control it or test it.
The simplest quick-fix is to start a separate thread or fork a subprocess to send the email. Yes, there likely will arise problems from using Gmail, and I really have no input on that vs. the alternatives. But from a design perspective, there's just no reason to make the user wait for that process to complete.
From a testing perspective, this might be where a proxy pattern might come in handy. It might be easy for you to directly invoke Gmail to send a message. Make it harder. Put in a proxy object that does the mailing for you that you can turn off (because heaven knows you can't for testing purposes make Gmail crash). Just make your team follow what happens in the event of an email malfunction by turning off the proxy and trying to complete an order. If you are doing it synchronously, then all the plagues mentioned here by other posters will rear their heads. If you are doing it asynchronously, you should be able to allow it to fail silently (from the user's perspective--from your perspective there should be enormous logging statements and text messages in the middle of the night and possibly a mild electric current arcing across the surface of someone's skin).