I'm building a simple webmail and I would like to make use of the CONDSTORE extension for IMAP: it allows to fetch everything that has changed (messages, flags, …) since a date, which is very practical to synchronize the email client to the IMAP server.
However I have absolutely no idea if this IMAP extension is actually supported by most IMAP servers or not.
So is there any kind of source that could tell me how well this extension is supported? For example, does GMail or Hotmail support it?
(the hidden question behind this is obvious: is CONDSTORE the most appropriate way to synchronize my email client to the IMAP server? I store locally copies of emails, that's why I need synchronization)
As I needed a more precise answer, I did a small node.js script and used some test accounts.
Here are the results, if you need more accounts, tell me.
yahoo
No CONDSTORE.
IMAP4REV1
ID
NAMESPACE
X-ID-ACLID
UIDPLUS
LITERAL+
XAPPLEPUSHSERVICE
XYMHIGHESTMODSEQ
gmail
CONDSTORE supported.
IMAP4REV1
UNSELECT
IDLE
NAMESPACE
QUOTA
ID
XLIST
CHILDREN
X-GM-EXT-1
UIDPLUS
COMPRESS=DEFLATE
ENABLE
MOVE
CONDSTORE
ESEARCH
outlook.com
No CONDSTORE.
IMAP4REV1
CHILDREN
ID
NAMESPACE
UIDPLUS
UNSELECT
No, it is not widely supported. You can treat it as an optimization to use when available, but you cannot depend on it. You also need a way to track removed messages, which is not something condstore can tell you
Gmail, however, just rolled out support for it recently.
Dovecot and gmail support condstore, and any marketing professional will tell you that they alone cover 70% of the users. Whether you believe marketing math is another question, and whether 70% is a lot in your eyes is a third, and neither of those questions are appropriate for SO.
To check any particular server, do either openssl s_client -connect asdf.example.org:993 -crlf or telnet asdf.example.org 143 to connect, then a login asdf fdsa to log in and finally b capability to get the list of extensions. You will see condstore listed in the reply. Or not, as the case may be.
Edit: Due to #guettli's request I took a convenient set of email addresses now, and for each address I determined a few possible IMAP server names heuristically and tried to connect to port 993. I see >80% CONDSTORE now, if I count only the addresses for whom I could connect, and do a little bit of guesswork, such as assuming that all Dovecot servers are new enough to support CONDSTORE.
However, your address mix isn't like mine (which happens to be dominated by one particular freemail provider), and the way to count depends on your purpose. Do you want to count the big providers that have CONDSTORE, but whose users mostly use the web interface?
If you want to count servers rather than users, Shodan can help. Searches for imap generally and for the most popular server says 75% of servers have condstore and the remaining 25% may or may not. But of course that's a misleading way to count: most users are on a few freemail services and most of the rest are on just a few thousand other providers.
If you want to run a survey yourself, connect to ports 143 and 993 and send a capability and then CRLF. Condstore is supported if the response includes either condstore, qresync, anything to do with modseq, or if the server banner names Dovecot or Cyrus. (Some servers won't tell you whether they support condstore until you have logged in, so looking at the server name is a good proxy.)
So is there any kind of source that could tell me how well this extension is supported? For example, does GMail or Hotmail support it?
For gmail:
$ openssl s_client -connect imap.gmail.com:993 -crlf
t login chetsteadman787#gmail.com ********
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH
Left as an exercise for other IMAP servers.
Related
Using the Smooch API, I am trying to obtain all of the messages sent to my Facebook appid in the past few minutes or hours.
The Get Messages REST method does exactly what I need, except for that it only returns messages from a particular appUserId. This isn't useful unless you already know what users have sent you messages. I cannot use a webhook as the application resides behind a corporate firewall. Opening the firewall to connections that originate from the outside is not an option (even with white-listing).
Is there a way to invoke the Get Messages REST method such that it will ignore the appUserId filter? Perhaps some sort of wildcard character?
GET {{url}}/{{apiVersion}}/apps/{{appId}}/appusers/{{appUserId}}/messages
Unfortunately you do need to have the appUserId (or userId) on hand in order to query user messages.
Webhooks are a pretty essential part of building a Smooch integration. If you can't receive them through your firewall, then you might consider building an intermediary service outside of your corporate network for receiving Smooch webhooks. For each webhook event you receive it would either:
Forward it through a secure tunnel into your coprorate network
Store the appUserId (or the whole event) in its own database, and provide a secure endpoint that allows your corporate network service to query that data
I'm curious to know more about your use case, e.g which Smooch channels are you integrating? With more details I might be able to improve this answer.
#alavers We would like to leverage nearly every messaging integration you offer.
#alavers You may want to consider providing a Get Messages variant that is better suited for use within a corporate firewall environment. An excellent example is the http long poll implementation provided by APIs such Amazon's SQS API. Their receiveMessage method waits for up to the specified time period but returns as soon as a message is received. This provides nearly the same performance of a webhook but eliminates the need for a customer to open their corporate firewall to connections that originate from outside the corporation. Most IT departments will approve connections that originate from within the corporation, but permitting connections that originate from the outside becomes a very difficult sell.
I'm trying to implement a webmail in PHP. I would like to write a PHP CLI script which is run on every email arrival to store some parts of (not all of) incoming email into database for search purposes. Then when the user finished searching and chose an email to show, a connection is made to mail server to retrieve the complete email. In order to implement this scenario I need to make some sort of connection among emails within database and mail server.
Since my knowledge of working with mail servers is limited to Zend Framework's API, what I believe I need in order to retrieve an email from an IMAP server is a message number or a message unique id (this later one seems not to be supported by all mail servers).
To this point, I've managed to find .forward (and some other ways) to introduce my PHP CLI script to MTAs to be run on every email arrival. This way I can store emails to database. But this won't do since message unique id is created by MDA so MTA do not know of it and they can not provide it to me. This means I can not find emails later when I want to retrieve them from mail server.
At last, here's my question: Is there a way to introduce a PHP CLI script to a MDA for emails' arrival? If this is dependent on the mail server, which servers do support this and how? My personal choice would be Dovecot or Courier, but any other mail server would do as well.
This is tricky -- there are many ways on how to setup delivery. Some of them work with the underlying mail store directly, bypassing your IMAP server altogether, while others use e.g. Dovecot's facilities.
Have you considered building on top of the notify plugin which ships with Dovecot?
It seems like it's impossible to introduce such a PHP CLI script to IMAP server (at least I'm sure of Dovecot). Anyway, the work around I found for this problem is to use my own PHP script to insert the new mails into IMAP server and retrieve their id's and then store the id in database for future references. To be clear, email are given to my PHP CLI script by MTA, not MDA. As I said before this is done easily using .forward file.
[UPDATE]
Unfortunately it seems this solution can not be implemented as well. The way to insert a new email to IMAP server is APPEND command, and to have the UID of the recently added mail server must support UIDPLUS extension. Neither Dovecot nor Courier supports this extension at the moment! If they did it seems the server would return the UID with a APPENDUID response.
[UPDATE]
It is my bad since Courier does support UIDPLUS. So this solution is valid and the one I'm going to implement.
In one of my Rails applications I'm sending emails through the Gmail SMTP server and everything just works, mails are not going to spam and so on ... But there's one thing that concerns me, the 500messages/day limit the SMTP has, currently I'm over 350/day. I didn't find any official Google page where they talk about the subject, just blog posts that seems to be reliable. Then my question is what do you suggest me in order to be able to send more than 500messages/day? I would love to keep using the Gmail SMTP.
Any help would be appreciated.
Don't use GMail for what it wasn't built for. It wasn't designed as a mass-email system, although Google definitely has the firepower to do this.
Instead, perhaps use something like SendGrid to send your emails. SendGrid is designed for this and is just as easy (actually, probably easier) to set up with than GMail.
You can consider using more than one gmail account to access the smtp server, however you may have problems with ip limits (couldn't find anything on their docs about it). Another option is performing dns mx lookups yourself and reproducing your own smtp server by directly delivering the messages to the user's emails - but that can increase the odds of you being tagged as a spammer as your ip is not whitelisted as sender.
I think the best way is to create another gmail account and trying to reroute your connection to use it when one of the accounts reaches the daily limit. A vpn connection can solve that for you.
Is there a way to determine POP or IMAP server given the email address? I am building an application for non-technical users and I dont really want to bother them with asking their IMAP/POP servers. mail2web.com does this, but I am not sure how.
This is how Thunderbird does it
/**
18 * Try to guess the config, by:
19 * - guessing hostnames (pop3.<domain>, pop.<domain>, imap.<domain>,
20 * mail.<domain> etc.)
21 * - probing known ports (for IMAP, POP3 etc., with SSL, STARTTLS etc.)
22 * - opening a connection via the right protocol and checking the
23 * protocol-specific CAPABILITIES like that the server returns.
24 */
http://mxr.mozilla.org/comm-central/source/mailnews/base/prefs/content/accountcreation/guessConfig.js
Thunderbird 3 does it too.. I'd take a look at the source code.
I think it's just a lookup table though..
There is nothing in a standard that dictates a POP/IMAP server for a given domain. Only convention or, as Joril suggests, lookup tables can be used. SMTP servers are different as there is a functional requirement to send the mail onto the next stop. Pick up (via POP/IMAP) is an entirely local domain admin issue. Sorry.
I guess you could take the domain and build a server name such that fred#mymail.com becomes pop.mymail.com and imap.mymail.com. Or perhaps take it further and interrogate the MX records and perform similar substitutions. Then you could run through your list of candidate servers looking for a POP/IMAP response. Might be a bit dodgy on the security front though.
Cheers,
Dan
There's absolutely no way to do this correctly in general.
However, you can use tables of common mail providers to fill in defaults, and you could fill in smtp.example.com etc... but that will still fail in some simple cases, though, like my work system where everything is on mail.wherever.com on unusual port numbers. So in the end, the user must be able to override whatever you do.
If you really want it to be general, you're going to have to deal with certificates and EAP as well.
Thunderbird does a good job. It's method is described here https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration. Below is a excerpt of the documentation.
All the lookup mechanisms use the email address domain as base for the lookup. For example, for the email address fred#example.com , the lookup is performed as (in this order):
tb-install-dir/isp/example.com.xml on the harddisk
check for autoconfig.example.com
look up of "example.com" in the ISPDB
look up "MX example.com" in DNS, and for mx1.mail.hoster.com, look up "hoster.com" in the ISPDB
try to guess (imap.example.com, smtp.example.com etc.)
We may in the future add DNS SRV records as supported mechanism in the future, but we currently do not.
When a user completes an order at my online store, he gets an email confirmation.
Currently we're sending this email via Gmail (which we chose over sendmail for greater portability) after we authorize the user's credit card and before we show him a confirmation message (i.e., synchronously).
It's working fine in development, but I'm wondering if this will cause a problem in production. Will it require making the user wait too long? Will many simultaneous Gmail connections get us in trouble? Any other general caveats?
If sending the emails synchronously will be a problem, could someone recommend an asynchronous solution (is ar_mailer any good?)
The main issue I can think of is that Gmail limits the amount of email you can send daily, so if you get too many orders a day it might break.
As they say :
"In an effort to fight spam and
prevent abuse, Google will temporarily
disable your account if you send a
message to more than 500 recipients or
if you send a large number of
undeliverable messages. If you use a
POP or IMAP client (Microsoft Outlook
or Apple Mail, e.g.), you may only
send a message to 100 people at a
time. Your account should be
re-enabled within 24 hours. "
http://mail.google.com/support/bin/answer.py?hl=en&answer=22839
I would recommend using sendmail on your server in order to have greater control over what's going on and don't depend on another service, especially when sendmail is not really complicated to set up.
The internet is not as resilient as some people would have you believe, the link between you and GMail will break at some point or GMail will go offline causing the user to think that they have not paid sucessfully.
I would put some other queue in place, sendmail sounds acceptable and you can't create your site now for where it 'might' be hosted in the future.
Ryan
If the server waits for the email to be sent before giving the user any feedback, were there problems connecting to the mailserver (timeouts, server down etc) the user request would timeout too and he wouldn't be told anything about the status of his order, so I believe you should really do this asynchronously.
Also, you should check whether doing that is even allowed by GMail's TOS. If that's not the case, you may check if that's allowed if you purchase one of their subscriptions. Also, there's surely a limit to the number of outgoing emails you may send within a given timeframe so if you're expecting your online store to be successful, you may hit that limit and bump into some nasty issue. If you're not self-hosting the site, you should check whether your host offers email servers (several plans include them for free) as then using your host's ISP would be the most obvious choice.
FACT: Gmail crashes. Not often, but it happens, and you can't control it or test it.
The simplest quick-fix is to start a separate thread or fork a subprocess to send the email. Yes, there likely will arise problems from using Gmail, and I really have no input on that vs. the alternatives. But from a design perspective, there's just no reason to make the user wait for that process to complete.
From a testing perspective, this might be where a proxy pattern might come in handy. It might be easy for you to directly invoke Gmail to send a message. Make it harder. Put in a proxy object that does the mailing for you that you can turn off (because heaven knows you can't for testing purposes make Gmail crash). Just make your team follow what happens in the event of an email malfunction by turning off the proxy and trying to complete an order. If you are doing it synchronously, then all the plagues mentioned here by other posters will rear their heads. If you are doing it asynchronously, you should be able to allow it to fail silently (from the user's perspective--from your perspective there should be enormous logging statements and text messages in the middle of the night and possibly a mild electric current arcing across the surface of someone's skin).