What's the best general technique for creating an IMAP client and keeping its local message store in sync with the server?
I guess I'm looking for the right way to figure out what's changed in an IMAP folder on the server since the last time I checked, and download those changes, to persist them to my local database... This would include messages no longer in the folder (deleted or moved), new messages, and changed messages...
I guess new messages is easy, I can grab the highest UID i have for a folder and then find messages since that UID. I'm not so sure about detecting messages that were deleted or moved though, or changed (maybe some flags changed on a message).
Thanks!
For sync, probably you need each folder all messages UID and flags.
You can compare local cached UIDs to server returned, with this you can dedect new messages and deleted(
Probably you should use some kind of hastable for search/compare, this will speed up all.
Related
I have a script that downloads my emails locally using mbsync, and then processes any emails that are marked as unread (marking them read, once completed). To avoid lengthy downloads I set this to only download the most recent 50.
My email folder has 1000 emails, 4 that are unread (which are the most recent).
Today I've noticed that mbsync is now pulling the emails from oldest first, vs newest, which means no emails get processed. I haven't amended anything with my code, only updated an email filter, to grab another email subject to my folder. Reverting this did nothing.
Is there a setting I can use to pull newest first or if this is some habit with gmail that I am unaware of, where new labels mess things up?
Below is a copy of my mbsync file, which should be correct based on my googling.
SyncState *
MaildirStore local,discreps
Path DIR/discrepancy/
IMAPAccount discreps
AuthMechs LOGIN
Host smtp.gmail.com
User EMAIL
Pass PWORD
SSLType IMAPS
SSLVersions TLSv1.2
IMAPStore discreps
Account discreps
Channel discreps,allmail
Master :discreps:"discrepancies"
Slave :local,discreps:discrepancy_emails
Create Slave
MaxMessages 50
ExpireUnread yes
CopyArrivalDate yes
Group discreps
Channels discreps,allmail
Seems Gmail has some weird behaviour that causes new filters, to mess with mbsync. After a day the issue seemed to resolve itself, and occur again if I made any new filters that affected folders (again resolving itself after a day.
New to Twilio. Developing an IT alerting function with Twilio SMS/MMS API in Python. A postfix alias-executed program processes a message and sends essential data via Twilio MMS to designated recipients.
Media such as images are accessed through media_url property to Client.messages.create(), via a URL pointing to content that I must store and offer through my HTTP server.
I have verified that that is the case, so my question is:
How do I control access to those images so that only Twilio can access them, and only for the duration of the message sending process?
My current solution, which is a kludge, is for the postscript alias-executed program to write a list of media files associated with the message, and then write my own status_callback that erases the files in that list when I get a "delivered" status (or a certain time limit expires).
This is a problem because the media files are publicly accessible for however long it takes for the "delivered" status to arrive or for my timeout to occur.
I've tried various searches but no applicable security mechanism has presented itself.
I use Basic authentication and serve all my Twilio content from a dedicated directory which is password protected, Twilio seems quite happy to accept urls with inline username#password parameters.
I think Twilio publish a list of their IP address ranges somewhere too, so if you really want to lock your media directory down you could whitelist those and deny everything else access to that dir within your server config.
To delete them once they are processed I would probably write a basic script that is triggered by the Twilio status webhook and adds the filename of the image which can be deleted to a database table. I think you can pass some sort of verification tokens for Twilio to return with a callback for additional security.
Then run another script every few mins as a cron job (under a different user account with permission to delete files in your media dir) which reads the database, deletes any files listed from the directory and then clears the database ready for the next time.
Edit
Thinking about it you can probably delete the files as soon as Twilio has queued your message as I'm pretty sure they copy your media files to their server upon submission. These files are publicly accessible (but with names nobody is likely to guess). You can delete them with HTTP DELETE
I have been requested to make my app available off-line, which means storing the data collected via Api for use when no connection available. The problem is that when a new connection is made my local data may be out of date. Also, any changes made while off-line will need to update the server.
I'm aware of a method of syncing databases so that when new connection is made the data is automatically updated both ways. However, after browsing Google I've not found a definitive method of doing this.
Can anyone help point me in the right direction?
There should be a field like a time stamp to indicate last synced time. When ever connection is online go for a fetch validate against the timestamp and update the data in offline storage.
The same way, when you have updates while offline you can set some bool value to check whether data is synced or not and sync when you are online.
For those of you familiar with IMAP - If I retrieve a draft message (or any message for that matter), and I wish to update it / edit it, what commands should I use?
The only command i've come across is Append, which appears to only insert, meaning I would have to delete the previous draft from the mailbox?
IMAP is designed for server-side management of mailboxes, not for editing messages. So yes, you would have to retrieve the contents of the desired message (FETCH), then delete that message from the server (STORE a \Deleted flag on the message and then EXPUNGE deleted messages), and then upload the updated message to the server (APPEND). There are no IMAP commands for editing the contents of an existing message, only for updating flags related to existing messages (STORE).
Remy's answer is correct. On top of that, you could optimize the process a bit if the server supports the CATENATE extension via APPEND CATENATE (so that you could save yourself uploading the existing attachments, etc).
I would like to create an email client that can access multiple IMAP mailboxes. I'd also like a copy of all emails for processing. What is the best way to do this using IMAP commands?
Right now I have a script that iterates over the folders, FETCHing FLAGS on 1:* to see what's been read and if any previously read messages have been marked as new, then FETCH BODY.PEEK on all of the messages I don't have in my database. Is there a better way?
A better way would be to fetch UIDs of all messages (UID FETCH 1:* FLAGS), compare the resulting UID list with your database and then download any messages you don't have and remove any messages you have but the server doesn't (deleted by other IMAP clients or using a web interface, for example). This is the only reliable method to duplicate an IMAP folder, AFAIK.
(And don't forget to take UIDVALIDITY into account as well!)
Your original method would not work correctly if other IMAP clients were accessing the mailbox in addition to your application. In theory, it would work OK if you can stay connected to the IMAP server continuously, using NOOP and IDLE to check new and deleted messages, but this is never possible in practice - even GMail doesn't have 100% uptime :-)
An ultimate IMAP client would combine both these approaches.