I'm trying to work out how best to connect / thread a chain of emails. This seems like such a common problem that I was surprised that I couldn't easily locate information on how other people have dealt with it. The only thing I found was a post about JWZ threading which looked more concerned with parsing together a thread in one email. I was wondering if anyone could point to me some current solutions.
I'm using the thoughtbot griddler gem to process incoming emails into a model Message(s) and a separate model Contact(s), and I have a third model for storing replies, e.g. Reply.
My current thinking is to thread them by the unique contact and the subject line. But then again the subject line will change slightly. e.g. from "This subject" -> "Re: re: This subject" I could use regex to try parsing out "re:"s or I could use something like amatch to do string comparisons?
But then again, what to do about the same subject appearing for the same user 2 months later? Also add some logic regarding the current date so that threads only use recent emails. Then there might be something else useful stored in the email header itself?
User (by unique email address)
Unique Subject line (regex re: processing issues?)
Current date (emails must be date relative to each other)
Some other clues to look for in the email header?
I have i rough idea of how to do it, I'm just curious to see some current implementations, I just can't seem to find any.
Any pointers would be greatly appreciated!
Email threads are a linked list, the information in the headers contains enough information to reconstruct the list from its component parts.
Introspect the email headers and to look for some specific headers.
The key ones you'll use are Message-ID, In-Reply-To and References. These headers give you information about which message was replied to and what other ids matter to the email thread itself.
The easiest way to find information about the headers of an email is to open the 'Original Message' in gmail (from the more menu).
There is a new gem named Msgthr, which is an implementation JWZ's algorithm. It's not matching subjects, senders or dates, so it's not exactly what you're looking for, but I think it's a good start.
The neatest thing about Msgthr is that it's container-agnostic, hence you don't have to install requirements such as TMail, as in Frederik Dietz's ruby port. This also means it can be used for other types of communications.
Here's some sample code, given a list of messages, let's group them into threads:
thr = Msgthr.new
threads = {}
[1, 11, 12, 2, 21, 211].each{ |id| threads[id] = [id]}
my_add = lambda do |id, refs, msg|
thr.add(id, refs, msg) do |parent, child|
threads[child.mid] = threads[parent.mid]
end
end
# Create the following structure
# 1
# \
# | 1.1
# \
# 1.2
# 2
# \
# 2.1
# \
# 2.1.1
my_add.call(1, nil, '1')
my_add.call(11, [1], '1.1')
my_add.call(12, [1], '1.2')
my_add.call(2, nil, '2')
my_add.call(21, [2], '2.1')
my_add.call(211, [21], '2.1.1')
thr.thread!
thr.rootset.each do |cnt|
threads[cnt.mid][0] = cnt.msg
end
Disclosure: I'm one of the contributors to the gem.
Related
I am reading in a lot of emails every day using net::imap
https://ruby-doc.org/stdlib-2.5.1/libdoc/net/imap/rdoc/Net/IMAP.html
I cannot seem to find a way to add a colored category to a message.
For example
imap.search(["ON","26-APR-2021"]).each do |message_id|
# my code...
# at the end of the code
message_id.AddCatagory(blue)...
end
The method name would be add_category in well-written ruby. But that doesn't exist, because colours aren't a concept in IMAP. Flags do exist, and I know some clients use colours to represent flags... so the code you're probably looking for is something along the lines of
imap.uid_store(uids, "+flags", [:Blue])
You have to define Blue, of course. Or Important, Flagged or something. Flagged is used by many clients for vaguely important messages.
I want to integrate 2 JIRA instances through email, one uses Jira 4.2.1 and the other uses 4.3.3.
one instance has certain custom fields, another has certain custom fields, both of the JIRA instances has to interchange the issue details, updates of the issue, through email. i.e both has to be in sync.
For Example
1) if an issue is created in Instance 1, a mail will be triggered and using that email, Instance 2 will create an issue there.
2) Also, if there is a update for an issue in Insance1 then a mail will be triggered to Instance 2 which will update the same issue in Instance 2.
Hope it clears !!
If I got your intentions right, i believe that there is an easier way to do so using the Jira remote API. For example, you could easily write a Python script, using the XML-RPC library, comparing the two systems and updating them if needed.
The problem with the email method you suggested is that you could easily create an endless loop of issue creating...
First, create a custom field in both instances, and call it something like "Sync". This will be used to mark issues once we'll sync them.
Next, enable the RPC plugin.
Finally, write a script that will copy the issues via RPC, example:
#!/usr/bin/python
# Sample Python client accessing JIRA via XML-RPC. Methods requiring
# more than basic user-level access are commented out.
#
# Refer to the XML-RPC Javadoc to see what calls are available:
# http://docs.atlassian.com/software/jira/docs/api/rpc-jira-plugin/latest/com/atlassian/jira/rpc/xmlrpc/XmlRpcService.html
import xmlrpclib
s1 = xmlrpclib.ServerProxy('http://your.first.jira.url/rpc/xmlrpc')
auth1 = s1.jira1.login('user', 'password')
s2 = xmlrpclib.ServerProxy('http://your.second.jira.url/rpc/xmlrpc')
auth2 = s2.jira1.login('user', 'password')
# go trough all issues that appear in the next filter
filter = "10200"
issues = s1.jira1.getIssuesFromFilter(auth1, filter)
for issue in issues:
# read issues:
for customFields in issue['customFieldValues']:
if customFields['customfieldId'] == 'customfield_10412': # sync custome field
# cf exists , dont sync!
continue
# no sync field, sync now
proj = issue['project']
type = issue['type']
sum = issue['summary']
desc = issue['project']
newissue = s2.jira1.createIssue(auth2, { "project": issue['project'], "type": issue['type'], "summary": issue['summary'], "description": issue['description']})
print "Created %s/browse/%s" % (s.jira1.getServerInfo(auth)['baseUrl'], newissue['key'])
# mark issue as synced
s.jira1.updateIssue(auth, issue['key'], {"customfield_10412": ["yes"]})
The script wasn't tested but should work. You'll probably need to copy the rest of the fields you have, check out this link for more info. As well, this is just one way sync, you have to sync it the other way around as well.
So, you're in Github filing an issue and you refer to issue #31. Then, while writing this issue, you note that #johnmetta has suggested some possible solutions that he's working on. Then you hit "Submit New Issue" and when you do, "#31" and "#johnmetta" are links, and #johnmetta has been notified, and issue #31 has a notification that it has been referenced.
I realize that there are more than one technologies at work here (Javascript goodies, etc), but what I'm looking for are some examples of how to do this type of thing in the Rails world. It's an interestingly difficult subject to search for.
What I've come up with conceptually is:
Have some identifier, such as # or # that is reserved
Upon submission, search for that identifier in the appropriate attribute
Upon finding it, search for the appropriate model with a field matching what follows
Once finding that, replace that text string with a link
Optionally, do whatever necessary to notify the referenced object
That said, it seems like it's super simple (explicitly coded, assumes friendly_id).
def prettify_user_links(str, source):
result = str
str.scan(/(#\S+)+/).each do |mtch|
# Strip off whatever identifier we're using
search_string = mtch[0].gsub('#','')
# Search for the matching model in the appropriate table
user = User.find(search_string)
if user
# If we find a matching model, create some link text and link it
link_txt = "<a href=>'#{user.url}'>#{mtch}</a>"
result.gsub!(search_string, link_txt)
# Notification. Not sure how/where, maybe with a message bus, or something more brute-force like
Comment.create :user_id => user.id, :body => "You have been mentioned in #{link_to comment.excerpt, comment} by #{link_to comment.owner, owner}"
return result
That would be my first cut, but I feel there have to be much more elegant solutions.
An additional aspect to this question: How would you grab a snippit of surrounding text. The brute force way would be to search n words before and m words after that string and grab all of that, then grab that sub-string from the results and do the search. Still, seems like there'd be a more elegant solution.
What you've described is the basic way; anything else is not terribly more elegant. It's helpful to see it as two parts: one is on receipt of the comment (when you should do notifications) and the other is on display of the comment, when you should do linkification.
This allows you to keep the original comment in its original form, which is helpful.
Perhaps put an after_create (so notifications aren't sent on every edit) on the comment model (assuming a comment model that includes a 'body' field):
[edit: added contextual info]
after_create :notify_mentions
def notify_mentions
body.scan %r{(.{0,40})#(\w+)(.{0,20})} do |match|
username = match[1]
context = [match.first, match.last]
Notification.send(match, context, self) if User.exists?(:login => username)
end
end
I use \w+ in place of \S+ because people often say things like:
Hey #JohnMetta, how are you doing?
and \S+ will capture the , which might be wrong. Pulling the # out of the capture group lets me ignore it during notification.
The context in the above match groups consists of the 40 characters before and 20 characters after the matched username for your snippet. Adjust to taste.
Then when displaying the message, you essentially create a helper something like what you had:
def linkify(body)
body.gsub %r{#\w+} do |match|
link_to match, :controller => :users, :action => :show, :id => match
end
end
#gsub is awesome like that, in that it takes a block and replaces with the contents.
It's not a lot more elegant than what you had, but it should give a pretty decent result.
I have the following:
#users = User.all
User has several fields including email.
What I would like to be able to do is get a list of all the #users emails.
I tried:
#users.email.all but that errors w undefined
Ideas? Thanks
(by popular demand, posting as a real answer)
What I don't like about fl00r's solution is that it instantiates a new User object per record in the DB; which just doesn't scale. It's great for a table with just 10 emails in it, but once you start getting into the thousands you're going to run into problems, mostly with the memory consumption of Ruby.
One can get around this little problem by using connection.select_values on a model, and a little bit of ARel goodness:
User.connection.select_values(User.select("email").to_sql)
This will give you the straight strings of the email addresses from the database. No faffing about with user objects and will scale better than a straight User.select("email") query, but I wouldn't say it's the "best scale". There's probably better ways to do this that I am not aware of yet.
The point is: a String object will use way less memory than a User object and so you can have more of them. It's also a quicker query and doesn't go the long way about it (running the query, then mapping the values). Oh, and map would also take longer too.
If you're using Rails 2.3...
Then you'll have to construct the SQL manually, I'm sorry to say.
User.connection.select_values("SELECT email FROM users")
Just provides another example of the helpers that Rails 3 provides.
I still find the connection.select_values to be a valid way to go about this, but I recently found a default AR method that's built into Rails that will do this for you: pluck.
In your example, all that you would need to do is run:
User.pluck(:email)
The select_values approach can be faster on extremely large datasets, but that's because it doesn't typecast the returned values. E.g., boolean values will be returned how they are stored in the database (as 1's and 0's) and not as true | false.
The pluck method works with ARel, so you can daisy chain things:
User.order('created_at desc').limit(5).pluck(:email)
User.select(:email).map(&:email)
Just use:
User.select("email")
While I visit SO frequently, I only registered today. Unfortunately that means that I don't have enough of a reputation to leave comments on other people's answers.
Piggybacking on Ryan's answer above, you can extend ActiveRecord::Base to create a method that will allow you to use this throughout your code in a cleaner way.
Create a file in config/initializers (e.g., config/initializers/active_record.rb):
class ActiveRecord::Base
def self.selected_to_array
connection.select_values(self.scoped)
end
end
You can then chain this method at the end of your ARel declarations:
User.select('email').selected_to_array
User.select('email').where('id > ?', 5).limit(4).selected_to_array
Use this to get an array of all the e-mails:
#users.collect { |user| user.email }
# => ["test#example.com", "test2#example.com", ...]
Or a shorthand version:
#users.collect(&:email)
You should avoid using User.all.map(&:email) as it will create a lot of ActiveRecord objects which consume large amounts of memory, a good chunk of which will not be collected by Ruby's garbage collector. It's also CPU intensive.
If you simply want to collect only a few attributes from your database without sacrificing performance, high memory usage and cpu cycles, consider using Valium.
https://github.com/ernie/valium
Here's an example for getting all the emails from all the users in your database.
User.all[:email]
Or only for users that subscribed or whatever.
User.where(:subscribed => true)[:email].each do |email|
puts "Do something with #{email}"
end
Using User.all.map(&:email) is considered bad practice for the reasons mentioned above.
I am using Ruby on Rails 3 and a MYSQL database. I would like to retrieve a regex from the database and then use that value to validate email addresses.
I aim to not put the regex value in line in my RoR application code, but outside so that the value can be recalled for other usages and from other places.
In order to populate the database, I put in my 'RAILS_ROOT/db/seed.rb' the following:
Parameter.find_or_create(
:param_name => 'email_regex',
:param_value => "[a-z0-9!#\$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?",
)
Notice: in the 'seed.rb' file I edited a little bit the original regex from www.regular-expressions.info adding two \ just before $. Here it is the difference:
#original from www.regular-expressions.info
[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?
#edited by me
[a-z0-9!#\$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?
After run rake db:seed in the Terminal, in MYSQL database I have this value (without \ near $):
[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?
Then in my RoR application I use the regex this way:
def validate(string)
email_regex = Regexp.new(Parameter.find_by_param_name('email_regex').param_value)
if email_regex.match(string)
return true
else
return false
end
end
The problem using the above regex is that I can successfully validate also email addresses with double '#' or without the final part like these:
name#surname#gmail.com # Note the double '#'
test#gmail
Of course I would like to refuse those email addresses. So, how can I adjust that? Or, how can I get what I want?
I tried also to seed these regex:
#case 1
\A[a-z0-9!#\$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\Z
#case 2
\\A[a-z0-9!#\$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\Z
#case 3
^[a-z0-9!#\$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$
that in the MYSQL database become respectively:
#case 1
A[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?Z
#case 2
\A[a-z0-9!#\$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\Z
#case 3
^[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$
but also them don't work as expected.
UPDATE
Debugging I have
--- !ruby/regexp /[a-z0-9!#$%&'*+\/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+\/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?/
that means that just before of all / characters Ruby added a \ character. Can be that my problem? In the 'seed.rb' file I tryed to escape all / adding \ statements but the debug output is always the same.
There are so many things wrong on so many levels here…
Storing application configuration in your database isn't recommended; slower performance, potential catch 22s (like how do you configure your database, from your database), etc. Try something like SettingsLogic if you don't want to have to build your own singleton configuration or use an initializer.
Rails has built in validation functionality as a mixin that's automatically part of any models inheriting from ActiveRecord::Base. You should use it, rather than define your own validation routines, especially for basic cases like this.
You can actually have an email address with multiple # signs, provided the first is escaped with a backslash or the local portion of the address is quoted.
Why are you escaping $ characters in a character class where they have no special meaning?
Regular expressions are okay for a very basic validation of an email address to make sure you didn't get complete garbage data to pass off to your mail server, but they aren't the best way to verify an email address.
I suggest you have a good look at the validations guide at RubyOnRails.org.
You shouldn't reinvent this wheel. See http://lindsaar.net/2010/1/31/validates_rails_3_awesome_is_true for a standard way to validate email addresses in Rails 3.
If you do choose to reinvent the wheel, don't use a regular expression. The gory details of why this is a bad idea are explained in http://oreilly.com/catalog/9780596528126, along with a very, very complicated regular expression that almost does it.