I have a script that downloads my emails locally using mbsync, and then processes any emails that are marked as unread (marking them read, once completed). To avoid lengthy downloads I set this to only download the most recent 50.
My email folder has 1000 emails, 4 that are unread (which are the most recent).
Today I've noticed that mbsync is now pulling the emails from oldest first, vs newest, which means no emails get processed. I haven't amended anything with my code, only updated an email filter, to grab another email subject to my folder. Reverting this did nothing.
Is there a setting I can use to pull newest first or if this is some habit with gmail that I am unaware of, where new labels mess things up?
Below is a copy of my mbsync file, which should be correct based on my googling.
SyncState *
MaildirStore local,discreps
Path DIR/discrepancy/
IMAPAccount discreps
AuthMechs LOGIN
Host smtp.gmail.com
User EMAIL
Pass PWORD
SSLType IMAPS
SSLVersions TLSv1.2
IMAPStore discreps
Account discreps
Channel discreps,allmail
Master :discreps:"discrepancies"
Slave :local,discreps:discrepancy_emails
Create Slave
MaxMessages 50
ExpireUnread yes
CopyArrivalDate yes
Group discreps
Channels discreps,allmail
Seems Gmail has some weird behaviour that causes new filters, to mess with mbsync. After a day the issue seemed to resolve itself, and occur again if I made any new filters that affected folders (again resolving itself after a day.
Related
New to Twilio. Developing an IT alerting function with Twilio SMS/MMS API in Python. A postfix alias-executed program processes a message and sends essential data via Twilio MMS to designated recipients.
Media such as images are accessed through media_url property to Client.messages.create(), via a URL pointing to content that I must store and offer through my HTTP server.
I have verified that that is the case, so my question is:
How do I control access to those images so that only Twilio can access them, and only for the duration of the message sending process?
My current solution, which is a kludge, is for the postscript alias-executed program to write a list of media files associated with the message, and then write my own status_callback that erases the files in that list when I get a "delivered" status (or a certain time limit expires).
This is a problem because the media files are publicly accessible for however long it takes for the "delivered" status to arrive or for my timeout to occur.
I've tried various searches but no applicable security mechanism has presented itself.
I use Basic authentication and serve all my Twilio content from a dedicated directory which is password protected, Twilio seems quite happy to accept urls with inline username#password parameters.
I think Twilio publish a list of their IP address ranges somewhere too, so if you really want to lock your media directory down you could whitelist those and deny everything else access to that dir within your server config.
To delete them once they are processed I would probably write a basic script that is triggered by the Twilio status webhook and adds the filename of the image which can be deleted to a database table. I think you can pass some sort of verification tokens for Twilio to return with a callback for additional security.
Then run another script every few mins as a cron job (under a different user account with permission to delete files in your media dir) which reads the database, deletes any files listed from the directory and then clears the database ready for the next time.
Edit
Thinking about it you can probably delete the files as soon as Twilio has queued your message as I'm pretty sure they copy your media files to their server upon submission. These files are publicly accessible (but with names nobody is likely to guess). You can delete them with HTTP DELETE
I'm trying to implement a webmail in PHP. I would like to write a PHP CLI script which is run on every email arrival to store some parts of (not all of) incoming email into database for search purposes. Then when the user finished searching and chose an email to show, a connection is made to mail server to retrieve the complete email. In order to implement this scenario I need to make some sort of connection among emails within database and mail server.
Since my knowledge of working with mail servers is limited to Zend Framework's API, what I believe I need in order to retrieve an email from an IMAP server is a message number or a message unique id (this later one seems not to be supported by all mail servers).
To this point, I've managed to find .forward (and some other ways) to introduce my PHP CLI script to MTAs to be run on every email arrival. This way I can store emails to database. But this won't do since message unique id is created by MDA so MTA do not know of it and they can not provide it to me. This means I can not find emails later when I want to retrieve them from mail server.
At last, here's my question: Is there a way to introduce a PHP CLI script to a MDA for emails' arrival? If this is dependent on the mail server, which servers do support this and how? My personal choice would be Dovecot or Courier, but any other mail server would do as well.
This is tricky -- there are many ways on how to setup delivery. Some of them work with the underlying mail store directly, bypassing your IMAP server altogether, while others use e.g. Dovecot's facilities.
Have you considered building on top of the notify plugin which ships with Dovecot?
It seems like it's impossible to introduce such a PHP CLI script to IMAP server (at least I'm sure of Dovecot). Anyway, the work around I found for this problem is to use my own PHP script to insert the new mails into IMAP server and retrieve their id's and then store the id in database for future references. To be clear, email are given to my PHP CLI script by MTA, not MDA. As I said before this is done easily using .forward file.
[UPDATE]
Unfortunately it seems this solution can not be implemented as well. The way to insert a new email to IMAP server is APPEND command, and to have the UID of the recently added mail server must support UIDPLUS extension. Neither Dovecot nor Courier supports this extension at the moment! If they did it seems the server would return the UID with a APPENDUID response.
[UPDATE]
It is my bad since Courier does support UIDPLUS. So this solution is valid and the one I'm going to implement.
What's the best general technique for creating an IMAP client and keeping its local message store in sync with the server?
I guess I'm looking for the right way to figure out what's changed in an IMAP folder on the server since the last time I checked, and download those changes, to persist them to my local database... This would include messages no longer in the folder (deleted or moved), new messages, and changed messages...
I guess new messages is easy, I can grab the highest UID i have for a folder and then find messages since that UID. I'm not so sure about detecting messages that were deleted or moved though, or changed (maybe some flags changed on a message).
Thanks!
For sync, probably you need each folder all messages UID and flags.
You can compare local cached UIDs to server returned, with this you can dedect new messages and deleted(
Probably you should use some kind of hastable for search/compare, this will speed up all.
I would like to create an email client that can access multiple IMAP mailboxes. I'd also like a copy of all emails for processing. What is the best way to do this using IMAP commands?
Right now I have a script that iterates over the folders, FETCHing FLAGS on 1:* to see what's been read and if any previously read messages have been marked as new, then FETCH BODY.PEEK on all of the messages I don't have in my database. Is there a better way?
A better way would be to fetch UIDs of all messages (UID FETCH 1:* FLAGS), compare the resulting UID list with your database and then download any messages you don't have and remove any messages you have but the server doesn't (deleted by other IMAP clients or using a web interface, for example). This is the only reliable method to duplicate an IMAP folder, AFAIK.
(And don't forget to take UIDVALIDITY into account as well!)
Your original method would not work correctly if other IMAP clients were accessing the mailbox in addition to your application. In theory, it would work OK if you can stay connected to the IMAP server continuously, using NOOP and IDLE to check new and deleted messages, but this is never possible in practice - even GMail doesn't have 100% uptime :-)
An ultimate IMAP client would combine both these approaches.
I am writing a client application that fetches emails from an IMAP server and then stores them in a database. The problem is that once I have checked the mail, the next time I only want to download the mail that has arrived since. So if I had checked the server for mail two hour ago, I only want to get the mail that has arrived in the last two hours.
I could use SEARCH with SINCE DATE, but there's no support for time + date could be easily spoofed.
I also tried the RECENT flag, but that doesn't seem to work with gmail (in ruby it shows nil everytime).
You want to use the UniqueId (UID) for the messages. This is specifically why it was created.
You will want to keep track of the last UID requested, and then, to request all new messages you use the message set "[UID]:*", where [UID] is the actual UID value.
For example, lets say the last message feteched had a unique id of "123456". You would fetch
123456:*
Then, discard the first returned message.
UIDs are 'supposed' to be stable across sessions, and never change, and always increase in value. The catch to verify this, is to check the UIDValidity when you select the folder. If the UIDValidity number hasn't changed, then the UIDs should still be valid across sessions.
Here are the relevant parts from the RFC:
2.3.1.1. Unique Identifier (UID) Message Attribute
A 32-bit value assigned to each message, which when used with the
unique identifier validity value (see below) forms a 64-bit value
that MUST NOT refer to any other message in the mailbox or any
subsequent mailbox with the same name forever. Unique identifiers
are assigned in a strictly ascending fashion in the mailbox; as each
message is added to the mailbox it is assigned a higher UID than the
message(s) which were added previously. Unlike message sequence
numbers, unique identifiers are not necessarily contiguous.
The unique identifier of a message MUST NOT change during the
session, and SHOULD NOT change between sessions. Any change of
unique identifiers between sessions MUST be detectable using the
UIDVALIDITY mechanism discussed below. Persistent unique identifiers
are required for a client to resynchronize its state from a previous
session with the server (e.g., disconnected or offline access
clients); this is discussed further in [IMAP-DISC].
Note: The next unique identifier value is intended to
provide a means for a client to determine whether any
messages have been delivered to the mailbox since the
previous time it checked this value.
Here is the link with more info:
http://www.faqs.org/rfcs/rfc3501.html
What I would do, is also keep track of the InternalDate of the messages downloaded. This way, if you ever lose UID sync, you can at least iterate through the messages, and find the last one you downloaded, based upon the InternalDate of the message.
There's an imap flag called "seen". Most clients would mark a message seen when viewing the message, so you'd want to iterate over messages on the server which do not have that flag set.
Here's a code snippet which should give you the right idea. The operative bit of course is
imap.search(["NOT", "SEEN"]).each do bla.bla.bla
If you are you able to filter incoming mail into a specific IMAP folder on the server side, your app
can read new messages in that folder and then move them into the standard INBOX folder after it's done.