I'm using the Mailman gem to process incoming email for my Rails app. My application looks for a YAML document in the plain-text email and then loads it into a Ruby object for further manipulation by the app.
However, I want to be able to plan ahead for email clients that might respond with a multi-part email. I need to get the plain-text part of the email and pass it into the YAML parser.
For some reason, it's still having problems parsing the YAML. I'm guessing because it's not really getting the plain text part here.
Is there a better way to get the text/plain part of an email with Mailman? Should I scrap Mailman and just get down and dirty with ActionMailer instead?
Mailman::Application.run do
default do
begin
message.parts.each do |part|
Mailman.logger.info part.content_type
if part.content_type == 'text/plain; charset=ISO-8859-1' # My poor way of getting the text part
the_yaml = part.body.decoded.scan(/(\-\-\-.*\.\.\.)/m).first.last # Find the YAML doc in the email and assign it to the_yaml
ruby_obj = YAML::load(the_yaml.sub(">", "")) # Remove any >'s automatically added by email clients
if ruby_obj['Jackpots']
ruby_obj['Jackpots'].each do |jackpot|
jp = Jackpot.find(jackpot['jackpot']['id'])
jp.prize = jackpot['jackpot']['prize']
jp.save
end
end
end
end
rescue Exception => e
Mailman.logger.error "Exception occurred while receiving message:\n#{message}"
Mailman.logger.error [e, *e.backtrace].join("\n")
end
end
end
I was able to find a little bit better way to handle getting the text part of the email.
Mailman::Application.run do
default do
begin
if message.multipart?
the_message = message.text_part.body.decoded
else
the_message = message.body.decoded
end
the_yaml = the_message.sub(">", "").scan(/(\-\-\-.*\.\.\.)/m).first.last
ruby_obj = YAML::load(the_yaml)
if ruby_obj['Jackpots']
ruby_obj['Jackpots'].each do |jackpot|
jp = Jackpot.find(jackpot['jackpot']['id'])
jp.prize = jackpot['jackpot']['prize']
jp.save
end
end
rescue Exception => e
Mailman.logger.error "Exception occurred while receiving message:\n#{message}"
Mailman.logger.error [e, *e.backtrace].join("\n")
end
end
end
And then after running it through the debugger and inspecting after the text part was successfully parsed. It would get hung up on the YAML loading. Turns out, a couple of my lines were too long, to the email client inserted a newline, breaking a comment in my YAML, and thus breaking the whole YAML document.
Related
I am sending attachments from Redmine to JIRA. This is the code I'm using for it:
if attachments.present?
begin
curl = SynchCurl.get_jira_curl_instance "jira_url"
curl.multipart_form_post = true
curl.headers["X-Atlassian-Token"] = "no-check"
attachments.each do |sa|
curl.http_post(Curl::PostField.file('file', sa.disk_filename, remote_file_name = sa.filename))
end
rescue Exception => e
CommonSynch.manage_jira_errors e
end
end
Now I want to check if the attachments are being saved on JIRA via Easy Callbacks to show the user an error message. I tried with:
curl.on_failure {|easy| puts "Error message"}
And some modifications under the post request, but the callback is always getting a nil value (I modified my code to provoke a 500 response code).
How can I do this?
Sorry if this is a dumb question, im pretty new in RoR. Thanks in advance.
What I want to achieve is the following:
Send an email with delayed_job containing:
plain-text
html (will be displayed by regular clients which don't understand the inline ical)
"inline" ical which is recognized by Outlook and Thunderbird (with Lightning).
a "regular" ical attachment (for #2)
What works so far/what does'nt:
I am able to send the email via delayed_job with all parts, however:
in Apple's Mail 2 attachments show up (instead of one):
(the html is displayed fine)
in Thunderbird (Lightning) I do get an invitation, just like I want. But the Alarm does not show up.
I have to do some REALLY disgusting gsubs on the rendered iCal in order for the ATTENDEES to show up. (see code snippet)
My thinking:
The first thing to keep in mind is: in order to send an email with attachments from delayed_job
To fix this, remember to add this line to your mailer: content_type "multipart/mixed"
As far as I understand the correct MIME-Type hierarchy would therefore be:
multipart/mixed
multipart/alternative
text/plain
text/html
text/calendar (with: method=REQUEST)
application/ics
Warning! code incoming.
I currently construct this email in the following manner:
Edit: I updated the mailer for Rails 4.2 (attachments must be placed before mail)
in my mailer.rb
def invitation_email(...)
subject = "I suck at email..."
attachments["invite.ics"] = { mime_type: "application/ics",
content: ical_attachment }
email = mail(from: me, to: you, subject: subject)
add_ical_part_to(email)
email
end
def add_ical_part_to(mail)
outlook_body = ical_attachment
mail.add_part(Mail::Part.new do
content_type "text/calendar; method=REQUEST"
body outlook_body
end)
end
and this is how I construct the ical attachments:
def ical_attachment
params_participant = {
"ROLE" => "REQ-PARTICIPANT",
"RSVP" => "FALSE",
"PARTSTAT" => "ACCEPTED"
}
params_invite = {
"CUTYPE" => 'INDIVIDUAL',
"ROLE" => "REQ-PARTICIPANT",
"PARTSTAT" => "NEEDS-ACTION",
"RSVP" => "TRUE"
}
cal = Icalendar::Calendar.new
event = Icalendar::Event.new
event.dtstart #party.from.to_datetime, { "VALUE" => "DATE" }
event.dtend #party.to.to_datetime, { "VALUE" => "DATE" }
event.summary #party.title
event.description #party.description
event.klass "PRIVATE"
event.organizer "cn=#{#user.name} #{#user.surname}:mailto:#{#user.email}"
# THIS DOES NOT WORK
event.alarm.trigger = "-PT5M" # 5 Minutes before...
#party.participations.each do |participation|
str = "cn=#{participation.user.name} #{participation.user.surname}:mailto:#{participation.user.email}"
event.add_attendee(str, params_participant)
end
#party.invitations.each do |invitee|
event.add_attendee("mailto:#{invitee.email}", params_invite)
end
cal.add_event(event)
cal.publish
# I KNOW THIS IS HORRIBLE AND I HATE IT, BUT OTHERWISE THE ATTENDEES DO NOT SHOW UP
cal.to_ical.gsub("ORGANIZER:", "ORGANIZER;").gsub("ACCEPTED:", "ACCEPTED;").gsub("TRUE:", "TRUE;").gsub("PUBLISH", "REQUEST")
end
Any help would be really appreciated!
The email that is being generated: http://pastebin.com/patf05zd
Oh and I'm on:
Rails 3.2.13
The Icalendar gem I'm using
In case someone else happens to come across this, here is what I did:
Instead of the icalendar gem I now use ri_cal. Although I was skeptical because the last commit to that repo was 3 years ago, the google group was a very helpful resource.
Here is how I generate the ical attachment (both inline and normal), which seems to be working fine (although it obviously needs some refactoring :))
def to_ical
# this is horrible
klass = self
cal = RiCal.Calendar do
event = event do
organizer "CN=#{klass.user.name} #{klass.user.surname}:mailto:#{klass.user.email}"
summary klass.party.title
description klass.ical_description
dtstart klass.party.from.utc.to_datetime
dtend klass.party.to.utc.to_datetime
location "See url in description"
security_class klass.security_class
# this is horrible
h = self
klass.party.participations.each do |participation|
h.add_attendee klass.prepare_participant(participation)
end
klass.party.invitations.each do |invitee|
h.add_attendee klass.prepare_invitee(invitee.email)
end
unless klass.party.reminder == 0
alarm do
description "Alarm description"
trigger klass.convert_trigger # -PT1H
action "DISPLAY"
end
end
end
end
# THE HORROR
cal.to_s.gsub("ATTENDEE:", "ATTENDEE")
.gsub("ORGANIZER:", "ORGANIZER;")
.gsub("CALSCALE:GREGORIAN", "CALSCALE:GREGORIAN\nMETHOD:REQUEST\n")
end
The 2 Attachments in Apples Mail still show up, I don't think that can be fixed.
Your second B64 encoded attachment contains a lot of garbage towards the end (attendee field).
That would explain the Thunderbird issue.
Please note that some clients will ignore any alarm you may set on a REQUEST: As an organizer, you should not dictate when each attendee should be reminded of the meeting. That would be a rather rude thing to do.
Regarding the Apple iCal issue, there is not much you can do I'm afraid: Some clients want the ics within, some as an attachment so you have to provide both. Does it show the accept/decline panel on iCal ?
I have used the pop mailer for ruby (net/pop)
The problem I am having is that some of the emails are in HTML format is there a way to specify that I want plain text ?
Thanks, Alex
Emails can come in different formats. The most common is MIME which allows an email to contain multiple "parts". Commonly an HTML and a plain-text part. However, you can not control which parts the email actually contains. This can only the sender for obvious reasons.
You can however use ruby to get the plain text part if one is present or try to generate some representation of that from the HTML part.
The following condensed example to get the plain text part of an email is from the MailHandler model and the POP3 module of Redmine (licensed under GPLv2).
def plain_text_body(email)
parts = email.parts.collect {|c| (c.respond_to?(:parts) && !c.parts.empty?) ? c.parts : c}.flatten
if parts.empty?
parts << email
end
plain_text_part = parts.detect {|p| p.content_type == 'text/plain'}
if plain_text_part.nil?
# no text/plain part found, assuming html-only email
# strip html tags and remove doctype directive
plain_text_body = strip_tags(email.body.to_s)
plain_text_body.gsub! %r{^<!DOCTYPE .*$}, ''
else
plain_text_body = plain_text_part.body.to_s
end
plain_text_body.strip
end
pop = Net::POP3.APOP(true).new(host,port)
pop.start(username, password) do |pop_session|
if pop_session.mails.empty?
puts "No email to process"
else
puts "#{pop_session.mails.size} email(s) to process..."
pop_session.each_mail do |msg|
message = msg.pop
plain_text = plain_text_body(message)
#
# Now do something with the plain text body
#
end
end
end
I have a rails code that sends emails. Following is in my controller:
def create
#users = Users.find(:all)
#sub = params[:sub]
#body = params[:body]
#index = 0
#users.each {|i| index++; Notifier.deliver_notification(#users.email_address, #sub, #body, #users.unsubscribe_link);}
flash[:notice] = "Mail was sent to " + #index + " people"
end
I have the following in my Model
class Notifier < ActionMailer::Base
def notification(email, sub, content, link)
recipients email
from "my_email#example.com"
subject sub
body :content => recipient, :link => link
end
end
This all works fine. My Question is:
For example if there is an error in sending mail to one of the pople, even then my flash message will say. Mail was sent to X people
What can I do to ensure that #index gets incremented ONLY when mail is successfully sent?
The deliver_notification method should always return a TMail object regardless of success or failure. There is a raise_delivery_errors setting which will allow the mailer to raise exceptions if there's trouble, but you'll have to rescue these in your block and only increment on success.
Due to the way mail is delivered by ActionMailer, it's often the case you won't know if the message is successful or not. Email is usually queued and delivered at a point in time well beyond your method call, and most errors occur at this point due to any number of difficulties in delivery. It's only wildly malformed email addresses that will be rejected up front, or if the mail delivery mechanism is non-functional.
Edit: Added Exception Tracking
count = 0
#users.each do |user|
begin
Notifier.deliver_notification(
user.email_address,
#sub,
#body,
user.unsubscribe_link
)
count += 1
rescue => e
# Something went wrong, should probably store these and examine them, or
# at the very least use Rails.logger
end
end
flash[:notice] = "Mail was sent to #{count} people"
Your example used index++ which is not supported by Ruby. What you probably want is index += 1. You were also using the #users array directly instead of the individual elements.
You could ask ActionMailer to throw exceptions for you, and then only count those deliveries that don't result in an exception.
ActionMailer::Base.raise_delivery_errors = true
#users.each do |i|
begin
Notifier.deliver_notification(#users.email_address, #sub, #body, #users.unsubscribe_link)
index++
rescue Exception => e
# Do whatever you want with the failed deliveries here
end
end
I want to get data from this page:
http://www.canadapost.ca/cpotools/apps/track/personal/findByTrackNumber?trackingNumber=0656887000494793
But that page forwards to:
http://www.canadapost.ca/cpotools/apps/track/personal/findByTrackNumber?execution=eXs1
So, when I use open, from OpenUri, to try and fetch the data, it throws a RuntimeError error saying HTTP redirection loop:
I'm not really sure how to get that data after it redirects and throws that error.
You need a tool like Mechanize. From it's description:
The Mechanize library is used for
automating interaction with websites.
Mechanize automatically stores and
sends cookies, follows redirects, can
follow links, and submit forms. Form
fields can be populated and submitted.
Mechanize also keeps track of the
sites that you have visited as a
history.
which is exactly what you need. So,
sudo gem install mechanize
then
require 'mechanize'
agent = WWW::Mechanize.new
page = agent.get "http://www.canadapost.ca/cpotools/apps/track/personal/findByTrackNumber trackingNumber=0656887000494793"
page.content # Get the resulting page as a string
page.body # Get the body content of the resulting page as a string
page.search(".somecss") # Search for specific elements by XPath/CSS using nokogiri
and you're ready to rock 'n' roll.
The site seems to be doing some of the redirection logic with sessions. If you don't send back the session cookies they are sending on the first request you will end up in a redirect loop. IMHO it's a crappy implementation on their part.
However, I tried to pass the cookies back to them, but I didn't get it to work, so I can't be completely sure that that is all that's going on here.
While mechanize is a wonderful tool I prefer to "cook" my own thing.
If you are serious about parsing you can take a look at this code. It serves to crawl thousands of site on an international level everyday and as far as I have researched and tweaked there isn't a more stable approach to this that also allows you to highly customize later on your needs.
require "open-uri"
require "zlib"
require "nokogiri"
require "sanitize"
require "htmlentities"
require "readability"
def crawl(url_address)
self.errors = Array.new
begin
begin
url_address = URI.parse(url_address)
rescue URI::InvalidURIError
url_address = URI.decode(url_address)
url_address = URI.encode(url_address)
url_address = URI.parse(url_address)
end
url_address.normalize!
stream = ""
timeout(8) { stream = url_address.open(SHINSO_HEADERS) }
if stream.size > 0
url_crawled = URI.parse(stream.base_uri.to_s)
else
self.errors << "Server said status 200 OK but document file is zero bytes."
return
end
rescue Exception => exception
self.errors << exception
return
end
# extract information before html parsing
self.url_posted = url_address.to_s
self.url_parsed = url_crawled.to_s
self.url_host = url_crawled.host
self.status = stream.status
self.content_type = stream.content_type
self.content_encoding = stream.content_encoding
self.charset = stream.charset
if stream.content_encoding.include?('gzip')
document = Zlib::GzipReader.new(stream).read
elsif stream.content_encoding.include?('deflate')
document = Zlib::Deflate.new().deflate(stream).read
#elsif stream.content_encoding.include?('x-gzip') or
#elsif stream.content_encoding.include?('compress')
else
document = stream.read
end
self.charset_guess = CharGuess.guess(document)
if not self.charset_guess.blank? and (not self.charset_guess.downcase == 'utf-8' or not self.charset_guess.downcase == 'utf8')
document = Iconv.iconv("UTF-8", self.charset_guess, document).to_s
end
document = Nokogiri::HTML.parse(document,nil,"utf8")
document.xpath('//script').remove
document.xpath('//SCRIPT').remove
for item in document.xpath('//*[translate(#src, "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")]')
item.set_attribute('src',make_absolute_address(item['src']))
end
document = document.to_s.gsub(/<!--(.|\s)*?-->/,'')
self.content = Nokogiri::HTML.parse(document,nil,"utf8")
end