Email threading - parsing

I'm working at a helpdesk application where i have a standalone script that queries a mail server and parses the mail it finds there.
I'm facing the following issue: How do i figure it out in a reliable way what mail is in reply to what mail?
I could add something in the subject like "[ticket:21312]" and look for that but what if the user changes the subject? Is there another way? Can i do it by setting a custom mail header and look for that or the header will not be preserved between mail servers on reply back from user?
What about when i send a message from my application to a non existing user or a user that has quota full and his server replies back with the usual standard message "the mail daemon at .... could not ...." then the subject will also be modified and i can't place the message correctly as a reply to an existing mail.
How does gmail do it? There the messages are sorted perfectly in almost all of the cases.

in helpdesk email piping there are 3 basic methods:
a) include the id in the subject somewhere (works fine in practice)
b) have the id in the body somewhere
c) use an auto-generated email alias with the id, like "case-76236781980893#helpdesk.mycompany.com". that can easily be handled by something like procmail or a script to pick out the id.
gmail might use a combination of the subject, In-Reply-To header (may not be defined) (References and Original-Message-ID headers possibly as well), and various heuristics, which work very well, but of course not necessarily bulletproof, and slightly more involved to implement. something like nestscape's original threading algorithm perhaps. though some have reported that gmail doesn't use the In-Reply-To header and relies mostly on the subject (as in this post).

As you say custom headers might get lost and the subject might change. Use both. If one exists then you can identify the thread. I don't know of any better way to solve this.

If your message was sent with a Message-ID-Header any standards-conform mailer should add a In-Reply-To-Header referencing your Id. Additionally Referencesshould contain a list of all previous mails in this thread.
This works with most mail clients, to be safe for the bad clients you have to use the subject, the easy way is by adding the "[issue:123]" thing, a secondary fallback is to recognize the subject (after cutting of the "Re:" part in all the variations) for this it could help that you know most of your legitimate senders ...

Related

Link in email to process an ics invitation

I have implemented an invitation system in my application, that generates an ical attachment that is sent to the appropriate users.
Is it possible to add a link in the email body that would trigger the processing of the invitation ? (in addition to the extra buttons provided by some email clients)
This question mentions that for email attachments in the general case it's not possible. Is it still the case for icalendar attachments ?
Other questions suggest using a service like https://www.addevent.com/, but this seems to be more oriented toward "public events", whereas in my case I would need something quite private (only invitations between 2 people), that are expected to be generates quite often (should be able to scale up to 100/day without problem), or it it just me getting a bad impression ?
Dealt with this problem several years ago and there was no way you could force most email client to "process" (trigger "Add to calendar" menu).
However, what you want is just make some element in your email trigger that. You can do this by simply linking that element to .ical file on your server. The advantage is that you can attache some parameters to your link and generate that event file dynamically. Also, it scales pretty well.

process incoming mail and parse out original text

I have inherited a rails forum (Rails 2.3.2 I think) that alerts people of new posts/replies for the forums or threads they are watching.
To make it easier for people to answer to threads I would like to enable reply-to-post, similar to basecamp and a bunch of other forums and tools out there.
I would add a separator text (like "----add your reply above this line-----") in the original email.
I need to:
process incoming email
extract the new text (above the separator line) as plain text
ideally strip out text like "on ... forums#mysite.com wrote:" that is automatically added by some mail clients
identify the thread this email is referring to (either using the incoming address or the subject line)
identify the sender
post the content as new reply
Any suggestions on how to get started? Any good plugins for this?
I've seen many mentioning Mailman and Fetcher, are there any other and which one is the best for this little feature?
Thanks!

Jira: Creating Issues and Comments from Email with no subject

I'm looking for a way to control the way the "Creating Issues and Comments from Email" works. At the moment since we are using the built in system any received email that has no subject cause the service to fail with the message "Issue must have a summary. The mail message has an empty or no subject.". Every time this happens we have to process the email manually.
Do you know of any way around this?
EDIT
The emails are being sent by customers, so I can't enforce all of them to send a subject.
The problem is that the mail creation service won't process the email once it has no subject.. so setting default value won't help. The only way i can think of is rewriting the mail service. Does anyone knows how could i do that? i I found the original source code here, but not sure how to build and deploy it.
Anyway can think of any other way?
Thanks!!
I assume that you just added some service to Jira in admin interface which automatically serializes emails.
If you want to fully customize the default behaviour, you can write your own simple jira plugin.
See the Atlassian pages:
https://developer.atlassian.com/display/DOCS/Set+up+the+Atlassian+Plugin+SDK+and+Build+a+Project
https://developer.atlassian.com/display/JIRADEV/Component+Plugin+Module
Your atlassian-plugin.xml should include something like:
<component key="message-handler-factory" class="com.atlassian.jira.plugins.mail.internal.DefaultMessageHandlerFactory" public="true">
<interface>com.atlassian.jira.service.util.handler.MessageHandlerFactory</interface>
</component>
You can for example extend the CreateOrCommentHandler class.

Generating a new email address on the fly, but not really!

I have a blogging application. Once a blog-post is created by a user, it will be sent as an email to some of user's friends. I want a functionality where the friends will just reply to the email and the content of the email will go as comments for that particular blog-post.
One way to do this is to do something similar to what http://ohlife.com does. It basically creates a unique ID per user per day, has the reply-to attribute of the email set to post+{unique_id}#ohlife.com and probably parses this field to know which user is the email for, when it gets received. But it really has only 1 email address which is post#ohlife.com. The part after the "+" get's ignored by email servers. This also is applicable to gmail.
What I wanted to know, is whether this property is for particular email servers or is it universal? If it is not universal, is there is email server independent way of implementing this? I would not want this to be based on the email subject, as it's the trivial solution I know of.
it is depending on your mail server and how it is configured.. (although it is quite a standard) - for example in postfix:
recipient_delimiter = +
you could set it to anything you like .. i once configured it to be a dot so i can use it all over the web.. http://www.postfix.org/postconf.5.html#recipient_delimiter
but you could simply make it configurable in your application as well..
Besides using the email subject or address, one other easy way to accomplish this would be to just stick an identifier number at the bottom of the outgoing email's body. It would then come back to you in the quoted part of the response message. This is much less obtrusive than putting stuff in the subject or address, and if you're using HTML messages you can even make the code invisible.

Best practices on processing email sent to app specific address in rails?

We would like to implement a feature by which users could send an email to an application specific address and we will parse the message and take certain actions on it, similar to 37signals's backpack (and probably some of their other apps).
If anyone has done something similar, could you fill me in on how you did so? I'm unsure on how to, at a high-level, 'import' the email into the app so that I could process it.
Thank you.
I have recently implemented that exact functionality in rails. I would advice you to look at the 'Receive E-mail Reliably via POP or IMAP' in the the Advanced Rails Recipes book.
I've personally found that the best source for getting this up and running and it explains how to do far better than I can. Good luck which ever way you choose to do it :)
Here's how I fetch from a POP server:
require 'net/pop'
pop = Net::POP3.new('mail.yourdomain.com')
pop.start(account, password)
pop.each_mail do |m|
email = TMail::Mail.parse(m.pop)
email.base64_decode
OttoMailer.process_email_in(email, m.unique_id)
m.delete
end
pop.finish
Why not run a Ruby SMTP mailserver, which will receive the mails via port 25, and then you can parse/interpret etc. as you wish ?
(I say Ruby since that's how you've tagged your question)
An alternative solution is to run procmail (or similar), pattern match on the subject, and then invoke scripts (configured in the .procmailrc file). However that may not scale so well for large volumes of mail.
ActionMailer can receive e-mails as well as send them! I don't remember how this is done but if you look at the documentation you can see it there. But from memory the e-mail gets piped through procmail into a script in the script directory.
There seems to be a book on the subject as well.
Good luck! :)

Resources