Seems to me that one of the default behaviors (i.e. The 3rd attempt to get the message route: "Fallback to the provided parameters, if they work") of .message_route(...) is not working properly.
It is supposed to parse the subject for an expression like [ID] to be used as 'thread_id' to post the message. So far, so good.
The problem is that the message is being parsed (of course) in .message_parse(...) before getting to this point, which makes that, when getting the 'thread_id' this way, we end up with a unicode-string instead of the required #ID as integer or long.
Am I right or am I missing something here?
def message_route(self, cr, uid, message, model=None, thread_id=None,
custom_values=None, context=None):
[...]
3. Fallback to the ``model``, ``thread_id`` and ``custom_values``
provided.
[...]
# 3. Fallback to the provided parameters, if they work
if not thread_id:
# Legacy: fallback to matching [ID] in the Subject
match = tools.res_re.search(decode_header(message, 'Subject'))
thread_id = match and match.group(1)
assert thread_id and hasattr(model_pool, 'message_update') or hasattr(model_pool, 'message_new'), \
"No possible route found for incoming message with Message-Id %s. " \
"Create an appropriate mail.alias or force the destination model." % message_id
if thread_id and not model_pool.exists(cr, uid, thread_id):
_logger.warning('Received mail reply to missing document %s! Ignoring and creating new document instead for Message-Id %s',
thread_id, message_id)
thread_id = None
_logger.debug('Routing mail with Message-Id %s: fallback to model:%s, thread_id:%s, custom_values:%s, uid:%s',
message_id, model, thread_id, custom_values, uid)
return [(model, thread_id, custom_values, uid)]
Related
We have several JIRA issues which have over 1000 duplicated, bogus, spam-like comments. How can we quickly delete them?
Background:
We disabled a user in active directory (Exchange) but not JIRA, so JIRA kept trying to email them updates. The email server gave a bounce-back message, and JIRA dutifully logged it to the task, which caused it to send another update, and a feedback loop was born.
The messages have this format:
Delivery has failed to these recipients or groups:
mail#example.com<mail#example.com>
The e-mail address you entered couldn't be found. Please check the recipient's e-mail address and try to resend the message. If the problem continues, please contact your helpdesk.
Diagnostic information for administrators:
Generating server: emailserver.example.com
user#example.com
#550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ##
Original message headers:
Received: from jiraserver.example.com (10.0.0.999) by emailserver.example.com (10.0.0.999)
with Microsoft SMTP Server id nn.n.nnn.n; Mon, 13 Jun 2016 15:57:04 -0500
Date: Mon, 13 Jun 2016 15:57:03 -0500
Our research did not discover an easy way without using purchased plug-ins such as Script Runner or "hacking" the database, which we wanted to avoid.
Note:
We came up with a solution and are posting here to share.
I created a python script to remove all comments for a specific Jira issue.
It uses the API from Jira.
'''
This script removes all comments from a specified jira issue
Please provide Jira-Issue-Key/Id, Jira-URL, Username, PAssword in the variables below.
'''
import sys
import json
import requests
import urllib3
# Jira Issue Key or Id where comments are deleted from
JIRA_ISSUE_KEY = 'TST-123'
# URL to Jira
URL_JIRA = 'https://jira.contoso.com'
# Username with enough rights to delete comments
JIRA_USERNAME = 'admin'
# Password to Jira User
JIRA_PASSWORD = 'S9ev6ZpQ4sy2VFH2_bjKKQAYRUlDfW7ujNnrIq9Lbn5w'
''' ----- ----- Do not change anything below ----- ----- '''
# Ignore SSL problem (certificate) - self signed
urllib3.disable_warnings()
# get issue comments:
# https://developer.atlassian.com/cloud/jira/platform/rest/#api-api-2-issue-issueIdOrKey-comment-get
URL_GET_COMMENT = '{0}/rest/api/latest/issue/{1}/comment'.format(URL_JIRA, JIRA_ISSUE_KEY)
# delete issue comment:
# https://developer.atlassian.com/cloud/jira/platform/rest/#api-api-2-issue-issueIdOrKey-comment-id-delete
URL_DELETE_COMMENT = '{0}/rest/api/2/issue/{1}/comment/{2}'
def user_yesno():
''' Asks user for input yes or no, responds with boolean '''
allowed_response_yes = {'yes', 'y'}
allowed_response_no = {'no', 'n'}
user_response = input().lower()
if user_response in allowed_response_yes:
return True
elif user_response in allowed_response_no:
return False
else:
sys.stdout.write("Please respond with 'yes' or 'no'")
return False
# get jira comments
RESPONSE = requests.get(URL_GET_COMMENT, verify=False, auth=(JIRA_USERNAME, JIRA_PASSWORD))
# check if http response is OK (200)
if RESPONSE.status_code != 200:
print('Exit-101: Could not connect to api [HTTP-Error: {0}]'.format(RESPONSE.status_code))
sys.exit(101)
# parse response to json
JSON_RESPONSE = json.loads(RESPONSE.text)
# get user confirmation to delete all comments for issue
print('You want to delete {0} comments for issue {1}? (yes/no)' \
.format(len(JSON_RESPONSE['comments']), JIRA_ISSUE_KEY))
if user_yesno():
for jira_comment in JSON_RESPONSE['comments']:
print('Deleting Jira comment {0}'.format(jira_comment['id']))
# send delete request
RESPONSE = requests.delete(
URL_DELETE_COMMENT.format(URL_JIRA, JIRA_ISSUE_KEY, jira_comment['id']),
verify=False, auth=(JIRA_USERNAME, JIRA_PASSWORD))
# check if http response is No Content (204)
if RESPONSE.status_code != 204:
print('Exit-102: Could not connect to api [HTTP-Error: {0}; {1}]' \
.format(RESPONSE.status_code, RESPONSE.text))
sys.exit(102)
else:
print('User abort script...')
source control: https://gist.github.com/fty4/151ee7070f2a3f9da2cfa9b1ee1c132d
Use the JIRA REST API through the Chrome JavaScript Console.
Background:
We didn't want to write a full application for what we hope is an isolated occurrence. We originally planned to use PowerShell's Invoke-WebRequest. However, authentication proved to be a challenge. The API supports Basic Authentication, though it's only recommended when using SSL, which we weren't using for our internal server. Also, our initial tests resulted in 401 errors (perhaps due to a bug).
However, the API also supports cookie-based authentication, so as long as you are generating the request from a browser which has a valid JIRA session, it just works. We chose that method.
Solution details:
First, find and review the relevant comment and issue IDs:
SELECT * FROM jira..jiraaction WHERE actiontype = 'comment' AND actionbody LIKE '%RESOLVER.ADR.RecipNotFound%';
This might be a slow query depending on the size of your JIRA data. It seems to be indexed on the issueid, so if you know that, specify it. Also, add other criteria to this query so that it only represents the comments you wish to delete.
The solution below is written for comments on a single issue, but with some additional JavaScript could be expanded to support multiple issues.
We need the list of comment IDs for use in the Chrome JavaScript console. A useful format is a comma-delimited list of strings, which you can create as follows:
SELECT '"' + CONVERT(VARCHAR(50),ID) + '", ' FROM jira..jiraaction WHERE actiontype = 'comment' AND actionbody LIKE '%RESOLVER.ADR.RecipNotFound%' AND issueid = #issueid FOR XML PATH('')
(This is not necessarily the best way to concatenate strings in SQL, but it's simple and works for this purpose.)
Now, open a new browser session and authenticate to your JIRA instance. We used Chrome, but any browser with a JavaScript console should do.
Take the string produced by that query and drop it in the JavaScript console inside of a statement like this:
CommentIDs = [StringFromSQL];
You will need to trim the trailing comma manually (or adjust the above query to do so for you). It will look like this:
CommentIDs = ["1234", "2345"];
When you run that command, you will have created a JavaScript array with all of those comment IDs.
Now we arrive at the meat of the technique. We will loop over the contents of that array and make a new AJAX call to the REST API using XMLHttpRequest (often abbreviated XHR). (There is also a jQuery option.)
for (let s of CommentIDs) {let r = new XMLHttpRequest; r.open("DELETE","http://jira.example.com/rest/api/2/issue/11111/comment/"+s,true); r.send();}
You must replace "11111" with the relevant issue ID. You can repeat this for multiple issue IDs, or you can build a multi-dimensional array and a fancier loop.
This is not elegant. It doesn't have any error handling, but you can monitor the progress using the Chrome JavaScript API.
I would use a jira-python script or a ScriptRunner groovy script. Even for a one-off bulk update, because it is easier to test and requires no database access.
Glad it worked for you though!
We solved this problem, which occurs from time to time, with ScriptRunner and a Groovy script:
// this script takes some time, when executing it in console, it takes a long time to repsonse, and then the console retunrs "null"
// - but it kepps running in the backgorund, give it some time - at least 1 second per comment and attachment to delete.
import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.issue.IssueManager
import com.atlassian.jira.issue.MutableIssue
import com.atlassian.jira.issue.comments.Comment
import com.atlassian.jira.issue.comments.CommentManager
import com.atlassian.jira.issue.attachment.Attachment
import com.atlassian.jira.issue.managers.DefaultAttachmentManager
import com.atlassian.jira.issue.AttachmentManager
import org.apache.log4j.Logger
import org.apache.log4j.Level
log.setLevel(Level.DEBUG)
// NRS-1959
def issueKeys = ['XS-8071', 'XS-8060', 'XS-8065', 'XRFS-26', 'NRNM-45']
def deleted_attachments = 0
def deleted_comments = 0
IssueManager issueManager = ComponentAccessor.issueManager
CommentManager commentManager = ComponentAccessor.commentManager
AttachmentManager attachmentManager = ComponentAccessor.attachmentManager
issueKeys.each{ issueKey ->
MutableIssue issue = issueManager.getIssueObject(issueKey)
List<Comment> comments = commentManager.getComments(issue)
comments.each {comment ->
if (comment.body.contains('550 5.1.1 The email account that you tried to reach does not exist')) {
log.info issueKey + " DELETE comment:"
//log.debug comment.body
commentManager.delete(comment)
deleted_comments++
} else {
log.info issueKey + " KEEP comment:"
log.debug comment.body
}
}
List<Attachment> attachments = attachmentManager.getAttachments(issue)
attachments.each {attachment ->
if (attachment.filename.equals('icon.png')) {
log.info issueKey + " DELETE attachment " + attachment.filename
attachmentManager.deleteAttachment(attachment)
deleted_attachments++
} else {
log.info issueKey + " KEEP attachment " + attachment.filename
}
}
}
log.info "${deleted_comments} deleted comments, and ${deleted_attachments} deleted attachments"
return "${deleted_comments} deleted comments, and ${deleted_attachments} deleted attachments"
I'm looking for an IMAP search clause to get GMail's "[Gmail]/All Mail" contents but filter out mails which are in "[Gmail]/Drafts".
IMAP DRAFT flag doesn't help with GMail's IMAP. By comparing full headers, I've noticed that drafts don't have value in "Delivered-To" field - it's missing from the header for all messages in "[Gmail]/Drafts".
I've tried UID search directly on "[Gmail]/Drafts" which has a few messages with following clause/results:
I'm expecting to get a clause which returns no messages - or all messages which have Delivered-To absent:
(HEADER Delivered-To "") <- returns all drafts
(NOT (HEADER Delivered-To "")) <- negating still returns all drafts!
(NOT HEADER Delivered-To "") <- still returns all drafts
(HEADER Delivered-To NIL) <- returns no messages - opposite to expected
(NOT (HEADER Delivered-To NIL)) <- returns all drafts
(NOT HEADER Delivered-To NIL) <- returns all drafts
(NOT (DRAFT)) <- returns all drafts
(DRAFT) <- returns none - opposite to expected, GMail doesn't seem to flag drafts with DRAFT
Will appreciate suggestions for how would you formulate "HEADER Delivered-To is not empty" for GMail IMAP.
x uid search header "delivered-to" "#"
The HEADER key searches for a header field with the given name containing the given substring. That is, header "foo" "" searches for messages that have a foo header field, empty or not. There is no proper way to search for messages that have a nonempty field.
However, you can get around that with a hack in this case: Every nonempty Delivered-To contains an # sign, so just search for # and you get the result you want.
Though not a direct answer, I ended up downloading bodies of new messages and scanning them for flags.
I'm trying to read email from GMail using gmail-xoauth Gem. I want to read the email and leave its unread status.
First, I tried reading just the header. Works.
imap = Net::IMAP.new('imap.gmail.com', 993, usessl = true, certs = nil, verify = false)
imap.authenticate('XOAUTH2', email, access_token)
imap.select('INBOX')
imap.search(["SINCE", since]).each do |message_id|
msg = imap.fetch(message_id,'RFC822.HEADER')[0].attr['RFC822.HEADER']
mail = Mail.read_from_string msg
puts mail.subject
end
Now, I want to read the body/text of the Email without marking it read.
This maybe be very late but I will leave it here for anyone else that stumbles onto this. If, for what ever reason you want to read the email and leave the flags intake, use:
imap.examine('INBOX')
instead of:
imap.select('INBOX')
From the Net::IMAP doc
Sends a EXAMINE command to select a mailbox so that messages in the mailbox can be accessed. Behaves the same as select(), except that the selected mailbox is identified as read-only.
Based on the documentation you need to use the store method. The documentation mentions:
store(set, attr, flags)
Sends a STORE command to alter data associated with messages in the mailbox, in particular their flags. The set parameter is a number or an array of numbers or a Range object. Each number is a message sequence number. attr is the name of a data item to store: ‘FLAGS’ means to replace the message’s flag list with the provided one; ‘+FLAGS’ means to add the provided flags; and ‘-FLAGS’ means to remove them. flags is a list of flags.
The return value is an array of Net::IMAP::FetchData. For example:
p imap.store(6..8, "+FLAGS", [:Deleted])
#=> [#<Net::IMAP::FetchData seqno=6, attr={"FLAGS"=>[:Seen, :Deleted]}>, \\
#<Net::IMAP::FetchData seqno=7, attr={"FLAGS"=>[:Seen, :Deleted]}>, \\
#<Net::IMAP::FetchData seqno=8, attr={"FLAGS"=>[:Seen, :Deleted]}>]
So you have to remove the :Seen flag
imap.store(message_id, "-FLAGS", [:Seen])
Anybody else seeing this?
There appears to have been some changes to the Google provisioning api for multi-domains. I have long running code that could restore a suspended user that has stopped working. I use Python and 2.0.17 of the Python GData libraries and the UpdateUser method to do this. I have also noted that RetrieveUser in the same library is no longer returning the first and last names of suspended users. I have filed an issue at Google apps-api-issues, please star if you are seeing this.
http://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3281
This is a simple example that will walk through the problem. Note that user_entry object returned from a RetrieveUser() on a suspended user will not have a property value for either first name or last name. The modified user_entry object is passed to UpdateUser() which does not allow the missing values for first and last names.
#!/usr/bin/python
import sys
import gdata.apps.multidomain.client
if len(sys.argv) < 4:
print "\nUsage:"
print sys.argv[0], "admin_email admin_password user_email\n"
sys.exit(0)
admin = sys.argv[1]
password = sys.argv[2]
email = sys.argv[3]
domain = ""
if '#' in admin:
admin_name,domain = admin.split('#', 1)
else:
print "Specify full email address of administrator.\n"
print "\nUsage:"
print sys.argv[0], "admin_email admin_password user_email\n"
sys.exit(0)
if '#' not in email:
print "Specify full email address of user.\n"
print "\nUsage:"
print sys.argv[0], "admin_email admin_password user_email\n"
sys.exit(0)
md_client = gdata.apps.multidomain.client.MultiDomainProvisioningClient(
domain=domain)
md_client.ClientLogin(email=admin, password=password, source='MDPROVISIONING')
print "Retrieve user: %s\n" %(email)
user_entry = md_client.RetrieveUser(email)
print user_entry
print ('\nRetrieve results: email: %s, suspended: %s,'
' first name: %s, last name: %s\n'
%(user_entry.email,user_entry.suspended,
user_entry.first_name,user_entry.last_name))
print "Update user (suspend): %s\n" %(email)
user_entry.suspended = 'true'
updated_user_entry = md_client.UpdateUser(email, user_entry)
print updated_user_entry
print ('\nSuspend results: email: %s, suspended: %s,'
' first name: %s, last name: %s\n'
%(updated_user_entry.email,updated_user_entry.suspended,
updated_user_entry.first_name,updated_user_entry.last_name))
print "Retrieve user: %s\n" %(email)
user_entry = md_client.RetrieveUser(email)
print user_entry
print ('\nRetrieve results: email: %s, suspended: %s,'
' first name: %s, last name: %s\n'
%(user_entry.email,user_entry.suspended,
user_entry.first_name,user_entry.last_name))
print "Update user (restore): %s\n" %(email)
user_entry.suspended = 'false'
updated_user_entry = md_client.UpdateUser(email, user_entry)
print updated_user_entry
print ('\nRestore results: email: %s, suspended: %s,'
' first name: %s, last name: %s\n'
%(updated_user_entry.email,updated_user_entry.suspended,
updated_user_entry.first_name,updated_user_entry.last_name))
Thanks user1803418, that explains why restoring a user works with GAM and not your code. The lack of firstName and lastName in the RetrieveUser API call is definitely an issue on Google's end that they'll need to fix. However, I'd describe the unsuspend issue as a client library deficiency that's excaberated by the firstName/lastName issue.
The API requires only attributes that are being updated be submitted in the update user API call. There's no need to specify the user's firstName, lastName and other object details. This means retrieving the user in order to perform the update simply makes the update process take longer, only one API call is really necessary.
GAM uses it's own implementation of the multidomain API calls which I wrote before the multidomain/client.py was released. GAM's implementation does not require that all user attributes be set in order to perform the update. You can see this by looking at:
http://code.google.com/p/google-apps-manager/source/browse/trunk/gdata/apps/multidomain/service.py#69
only the attributes that are set when calling UpdateUser() are sent to Google. You can see this for yourself by installing GAM and creating a file named debug.gam before unsuspending the user with the GAM command:
gam update user suspended off
the XML request to Google will be minimal, showing only a few fields including the suspended status of the user. firstName and lastName will not be included in the request.
So I think you have 3 options here:
1) wait for a fix from Google for the firstName/lastName issue. This should resolve the unsuspend issue also.
2) rewrite the multidomain/client.py UpdateUser() function so that it does not require all user attributes to be set in order to perform the update.
3) switch your code to use GAM's custom multidomain/service.py library since it supports updating only provided attributes for a user.
I hope that Google fixes the missing names soon.
Modifying UpdateUser in the multidomain client
(gdata.apps.multidomain.client) seems to keep things going:
def update_user(self, email, user_entry, **kwargs):
user_entry = gdata.apps.multidomain.data.UserEntry(
email=email,first_name=user_entry.first_name,
last_name=user_entry.last_name,password=user_entry.password,
change_password=user_entry.change_password_at_next_login,
suspended=user_entry.suspended, is_admin=user_entry.is_admin,
ip_whitelisted=user_entry.ip_whitelisted, quota=user_entry.quota)
return self.update(user_entry,
uri=self.MakeMultidomainUserProvisioningUri(email),
**kwargs)
I have a rails application which processes incoming emails via IMAP. Currently a method is used that searches the parts of a TMail object for a given content_type:
def self.search_parts_for_content_type(parts, content_type = 'text/html')
parts.each do |part|
if part.content_type == content_type
return part.body
else
if part.multipart?
if body = self.search_parts_for_content_type(part.parts, content_type)
return body
end
end
end
end
return false
end
These emails are generally in response to a html email it sent out in the first place. (The original outbound email is never the same.) The body text the method above returns contains the full history of the email and I would like to just parse out the reply text.
I'm wondering whether it's reasonable to place some '---please reply above this line---' text at the top of the mail as I have seen in a 37 signals application.
Is there another way to ignore the client specific additions to the email, other than write a multitude of regular expressions (which I haven't yet attempted) for each and every mail client? They all seem to tack on their own bit at the top of any replies.
I have to do email reply parsing on a project I'm working on right now. I ended up using pattern matching to identify the response part, so users wouldn't have to worry about where to insert their reply.
The good news is that the implementation really isn't too difficult. The hard part is just testing all the different email clients and services you want to support and figuring out how to identify each one. Generally, you can use either the message ID or the X-Mailer or Return-Path header to determine where an incoming email came from.
Here's a method that takes a TMail object and extracts the response part of the message and returns that along with the email client/service it was sent from. It assumes you have the original message's From: name and address in the constants FROM_NAME and FROM_ADDRESS.
def find_reply(email)
message_id = email.message_id('')
x_mailer = email.header_string('x-mailer')
# For optimization, this list could be sorted from most popular to least popular email client/service
rules = [
[ 'Gmail', lambda { message_id =~ /.+gmail\.com>\z/}, /^.*#{FROM_NAME}\s+<#{FROM_ADDRESS}>\s*wrote:.*$/ ],
[ 'Yahoo! Mail', lambda { message_id =~ /.+yahoo\.com>\z/}, /^_+\nFrom: #{FROM_NAME} <#{FROM_ADDRESS}>$/ ],
[ 'Microsoft Live Mail/Hotmail', lambda { email.header_string('return-path') =~ /<.+#(hotmail|live).com>/}, /^Date:.+\nSubject:.+\nFrom: #{FROM_ADDRESS}$/ ],
[ 'Outlook Express', lambda { x_mailer =~ /Microsoft Outlook Express/ }, /^----- Original Message -----$/ ],
[ 'Outlook', lambda { x_mailer =~ /Microsoft Office Outlook/ }, /^\s*_+\s*\nFrom: #{FROM_NAME}.*$/ ],
# TODO: other email clients/services
# Generic fallback
[ nil, lambda { true }, /^.*#{FROM_ADDRESS}.*$/ ]
]
# Default to using the whole body as the reply (maybe the user deleted the original message when they replied?)
notes = email.body
source = nil
# Try to detect which email service/client sent this message
rules.find do |r|
if r[1].call
# Try to extract the reply. If we find it, save it and cancel the search.
reply_match = email.body.match(r[2])
if reply_match
notes = email.body[0, reply_match.begin(0)]
source = r[0]
next true
end
end
end
[notes.strip, source]
end
I think you will be stuck on this one. I have been doing some stuff with emails myself in TMail recently, and what you will generally find is that an email that has an HTML part is generally structured like:
part 1 - multipart/mixed
sub part 1 - text/plain
sub part 2 - text/html
end
The email clients I have played with Outlook and Gmail both generate replies in this format, and they just generally quote the original email inline in the reply. At first I though that the 'old' parts of the original email would be separate parts, but they are actually not - the old part is just merged into the reply part.
You could search the part for a line that begins 'From: ' (as most clients generally place a header at the top of the original email text detailing who sent it etc), but its probably not guaranteed.
I don't really see anything wrong with a --- please reply above this line --- generally, its not that invasive, and could make things a lot simpler.