We use Microsoft Graph to subscribe to webhooks from emails. Additionally, as a backup procedure we also crawl the messages directly.
We crawl around 5 million emails a day, and we see that each day consistently around 1%-2% of these emails are not sent to us via the webhook, although the subscription for this principal is active (and other email notifications from this user are indeed sent).
Is there any logic on the Microsoft Graph side to not send webhooks for certain types of emails by design? or is it just a problem on the notification mechanism?
(Obviously crawling them, as we do now, is a viable backup option, but that also means the processing of the email will be delayed)
I currently have a similar webhook setup and we get around 200-300 emails and I notice that the subscription service usually misses out on 1-2 per day since sometimes some emails come around at the same time. I have also noticed that the data structure is an array of objects when we get two or more emails at the same time. What we have put into place is basically a cron scheduled script that checks the mailbox on specific time intervals, such as every 5 minutes, every 10 minutes and so on. This is the only solution that has worked for my application to capture every single email.
Related
I'm currently running a web application that uses Microsoft Graph's API and we encountered the following message today which severely impacted our application, for a whole day:
"error": {
"code": "ErrorTooManyObjectsOpened",
"message": "Too many concurrent connections opened., The process failed to get the correct properties.",
"innerError": {
"request-id": "removed",
"date": "2017-12-13T17:01:14"
}
}
please note that the request-id was removed
Let me summarize what our web application does.
Basically, we have 2 email folders that we are actively subscribed to, Junk and Folder A.
If anything hits Folder A, we strip the body of the email message and then move the message to Folder B. The subscription on our Junk folder also strips the body and sends them over to Folder B.
Sometimes the webhook subscription service skips messages that may come at the same time, therefore we have 2 cron jobs in our server that run a script and check Junk/Folder A for any messages every 5 minutes, therefore my assumption is that the cron job runs about 288*2 times per day. Not counting our subscription to the folders, we usually get around 200-300 email messages per day.
Unfortunately Microsoft's Graph error codes page does not provide us with any explanation about this code. I would really appreciate if anyone can explain what this means and how to avoid it from happening.
This is occurring because your application is exceeding the throttling thresholds.
There are several different throttling metrics that can affect Microsoft Graph requests. For a high-level overview, see the Microsoft Graph throttling guidance. Since in this case you're hitting Exchange Online via Graph, you can find more specific information from What throttling values do I need to take into consideration? in the Exchange documentation.
Architecturally, you are making a lot of unnecessary calls into the API. Rather than having both a subscription and a scheduled job, you should use just the webhook subscription and the /delta endpoint.
Each call to the /delta endpoint gives you a token that can be used to fetch any changes to a given resource since the token was originally issued. So regardless of if 1 email came in or 1,000, you only get the new emails.
Once you're using the /delta to find your changes, you then use a webhook only as a "trigger". When you receive the webhook, you can ignore the contents and instead issue a request to /delta. This ensures that you capture every incoming email even if you didn't necessarily receive separate webhook notifications.
There is a bug. After making 500 message move requests, a "cannot copy/move error" occurs. Subsequently, a "429: Too many concurrent connections opened" error occurs. Most applications miss the first error because you continually get the 429 error afterwards.
If you let the application "rest" for 30 minutes, the throttle resets itself and you can continue on. I do not think there is a time limit for hitting the 500 moves. My application did 500 moves after 6.5 hours and then we started getting the error.
And, if you keep trying your move call before the 30 min rest period, it never resets. Also, in the response, the retry-after is null... so, that doesn't help you.
If you find a work around, please let me know. We are trying a few things like setting the category, then manually moving the messages. I am also investigating making a rule the moves them for us or some other job. I cannot find a way to execute a rule from the Graph API.
See this link for more information. Also, the more people who report having this issue, hopefully the sooner it can be resolved. Outlook API Throttling documentation #144
I try to create a mailing list feature in Rails which is based on delayed_jobs. For now I send mails by iterating over a users table and .deliver a mail to every mail address.
how can i integrate it into delayed_jobs, so it sends 50 mails every 5 minutes and remembers which adresses are already done? do i need to make a seperate table where i store all sent mails and check back everytime i send another 50 mails?
thanks in advance.
You will probably want to have table entries for sent emails. That way it serves as an audit trail if processes go down or somehow fail.
Suggest you look at doing this with an elastic cloud database like MongoLab, MongoHQ or SimpleDB. (Mongo-based services make it easy to extend the schema for new email entries.)
If you do that, then a cloud worker queue like SimpleWorker can make it easy to send out lots of emails concurrently or in batches to get around any rate limits. (full disclosure: I work at Iron.io/SimpleWorker)
You're taking a good approach to bundle multiple email sends into a single worker task to amortize the worker setup costs. With an elastic cloud worker system, you could have master workers come off schedule and then queue up a number of slave worker tasks, each with a set of users to send.
With table entries, you can then go back through the data tables and address any emails that failed or didn't go through.
50 emails is not really so much can be sent in seconds, I think. Use foreverb for sending emails every 5 minutes.
Let delayed job do all the work:
User.all.each_with_index |user, index|
Mailer.delay({:run_at => ((index / 50) * 5).minutes.from_now}).send_newsletter(user)
end
This should work but untested.
I'm building a web application were users need to ability to send out mass emails. The application is a ticket site where individuals sell tickets to various events. In turn, events have promoters that sell tickets. The sponsor of the event needs to ability to send mass emails to their promoters. An event may have hundreds of promoters. So I'm assuming looping through each promoter and sending an email wont cut it.
Does rails 3.0 provide some kind of mass email functionality?
Thanks,
Brian
I don't think you should have a problem looping through all the promoters and sending each one an email. However, you would want to do that as a background process.
Check out delayed-job or http://www.simpleworker.com/. Both of these will let you load the email into a background job, and allow you to redirect the user in a timely manner while the emails get sent in the background.
Solomon is right, you'll definitely want to do this in a background task, which is pretty simple with something like delayed job. Have you thought about how the messages will actually be delivered though? What mail server you are going to use? Many providers won't let you send out hundreds of messages at a time, and you are likely to run into spam issues if you try to send that volume of messages from a personal email account. You may want to take a look at a service like Mailgun that specializes in this. From their FAQ:
Why not just use Sendmail + Postfix + Courier IMAP?
You can but you should be aware that there is a constant battle raging
between good and evil (i.e., spam) in the email universe. In order to
be on the 'good' side of that battle and get your email delivered
there are numerous things you need to do. You need to have the right
infrastructure and register your IP and Domain appropriately. Also,
you need to have a history of email sending that complies with ESPs
rules in order to build a good reputation.
Moreover, if you are going to receive, store and host emails, you
better be prepared for maintaining this orchestra of software, take
care of backups, hardware failures, security patches and monitoring.
Stop kidding yourself, it's not 1998 anymore.
I'm not affiliated with them in any way, but in my experience getting rails to send email is trivial compared to dealing with getting it delivered successfully by the mail server.
For my client's custom-built CRM, I want users (technicians) to be notified of changes to marked cases via email.
This warrants a simple subscription mapping table between users and cases and automated emails to be sent every time a change is made to a case from within the logging method.
How do I send 10-100 emails to subscribed users without bogging down my logging method? My SMTP server is on a peer on my LAN, so sends should be quick, but ideally this should be handled by an external queuing process.
I can have a cron job send any outstanding emails every 10 minutes, but for this specific client cases are quite time-sensitive and instant notification (as instant as email can be) would be great.
How can I send bulk notification emails from within ASP.NET MVC without bogging down my logging method?
Back in 2007 I was asked to look into a case where Web Server would suddenly freeze and start sending 503 errors and come back after a few minutes. Cutting a long story short it turned out at the end that it was sending email which was blocking the server (in addition to some bad code).
Basically Microsoft's SMTP server is implemented as a single-thread service - last I checked. This will mean that all your valuable ASP.NET threads serving requests will have to queue to a single thread monolithic application to send a not so urgent email. You need to decouple your web site from sending emails - that is what everyone does and there is a good reason for it.
Write your emails to a queue and have a process reading and sending emails.
Sending emails from an ASP.NET application is not a good idea as it might monopolize valuable server resources. A better solution would be to setup a Windows Service to perform this task or even write a Console application which could be scheduled to run with Windows Scheduler. Quartz.NET is a good solution you might take a look at allowing you to schedule jobs.
If you want those email sending to be triggered from within a particular controller action you could also have a separate WCF service which could be invoked asynchronously.
When a user completes an order at my online store, he gets an email confirmation.
Currently we're sending this email via Gmail (which we chose over sendmail for greater portability) after we authorize the user's credit card and before we show him a confirmation message (i.e., synchronously).
It's working fine in development, but I'm wondering if this will cause a problem in production. Will it require making the user wait too long? Will many simultaneous Gmail connections get us in trouble? Any other general caveats?
If sending the emails synchronously will be a problem, could someone recommend an asynchronous solution (is ar_mailer any good?)
The main issue I can think of is that Gmail limits the amount of email you can send daily, so if you get too many orders a day it might break.
As they say :
"In an effort to fight spam and
prevent abuse, Google will temporarily
disable your account if you send a
message to more than 500 recipients or
if you send a large number of
undeliverable messages. If you use a
POP or IMAP client (Microsoft Outlook
or Apple Mail, e.g.), you may only
send a message to 100 people at a
time. Your account should be
re-enabled within 24 hours. "
http://mail.google.com/support/bin/answer.py?hl=en&answer=22839
I would recommend using sendmail on your server in order to have greater control over what's going on and don't depend on another service, especially when sendmail is not really complicated to set up.
The internet is not as resilient as some people would have you believe, the link between you and GMail will break at some point or GMail will go offline causing the user to think that they have not paid sucessfully.
I would put some other queue in place, sendmail sounds acceptable and you can't create your site now for where it 'might' be hosted in the future.
Ryan
If the server waits for the email to be sent before giving the user any feedback, were there problems connecting to the mailserver (timeouts, server down etc) the user request would timeout too and he wouldn't be told anything about the status of his order, so I believe you should really do this asynchronously.
Also, you should check whether doing that is even allowed by GMail's TOS. If that's not the case, you may check if that's allowed if you purchase one of their subscriptions. Also, there's surely a limit to the number of outgoing emails you may send within a given timeframe so if you're expecting your online store to be successful, you may hit that limit and bump into some nasty issue. If you're not self-hosting the site, you should check whether your host offers email servers (several plans include them for free) as then using your host's ISP would be the most obvious choice.
FACT: Gmail crashes. Not often, but it happens, and you can't control it or test it.
The simplest quick-fix is to start a separate thread or fork a subprocess to send the email. Yes, there likely will arise problems from using Gmail, and I really have no input on that vs. the alternatives. But from a design perspective, there's just no reason to make the user wait for that process to complete.
From a testing perspective, this might be where a proxy pattern might come in handy. It might be easy for you to directly invoke Gmail to send a message. Make it harder. Put in a proxy object that does the mailing for you that you can turn off (because heaven knows you can't for testing purposes make Gmail crash). Just make your team follow what happens in the event of an email malfunction by turning off the proxy and trying to complete an order. If you are doing it synchronously, then all the plagues mentioned here by other posters will rear their heads. If you are doing it asynchronously, you should be able to allow it to fail silently (from the user's perspective--from your perspective there should be enormous logging statements and text messages in the middle of the night and possibly a mild electric current arcing across the surface of someone's skin).