Our platform is replicating email functionality, I.e. view all emails folders and contents, reply, create new, draft, move etc for users.
We have successfully subscribed users to all change notifications (create, update, & delete) for the whole mailbox, however, we are not sure how to track folder move operations as I can’t find an example in the documentation and our current implementation is not working reliably.
The issue we have is that after receiving the various change notifications, when we are doing the requests to get the value for the updated/deleted message, sometimes the value returned is the updated value, not the original one, therefore if the folder has changed we do not know which message to delete. This issue is highlighted in the documentation here (half way down): https://learn.microsoft.com/en-us/graph/outlook-change-notifications-overview?tabs=http#example-3-create-a-subscription-to-get-change-notifications-with-resource-data-for-a-message-based-on-a-condition-preview
We tried it using immutable ids, but the final webhook received was sometimes the delete webhook for the message ID. This is supposed to be for the original email that was moved, however because the message ID is the same (and the parent folder ID value is not reliable) we may end up deleting the wrong email.
With immutable turned off, we did not receive any delete notifications (only creates and updates), so we ended up with duplicate emails as the original was never deleted.
Is someone able to advise the correct procedure to track these events?
Thanks
So it turns out the only reliable way to do this is using the delta query - https://learn.microsoft.com/en-us/graph/delta-query-messages - whenever a change notification is received for a folder.
So, when authorisation is provided for access to a users mailbox, you must get subscriptions for each folder and then whenever a change notification is received for that folder/subscription, the delta query is run for that folder.
I believe that MS are in beta testing for providing the change information in the webhook which in my option would be a great improvement in efficiency in terms of implementation and operation.
Hope this helps someone in the future!
Related
I'm building a Slack application/bot and I want it to be able to welcome user every time one starts a direct message conversation with it.
For that purpose, I set up my Slack application accordingly with the documentation in order to be able to receive the 'im_created' event. Especially, I added the im:read bot scope (and even the im:read workspace scope, just to make sure) and subscribe to this event. But I never received it whereas I could receive all other type of events I needed.
Has someone already been in this case and share with me what I'm missing here ?
The 'im_created' event should not be relied upon, at least on small workspace. Indeed, this what the Slack support person answered to my request:
"I was initially testing on a workspace that had a lot of users. Then I decided to test on a smaller workspace and there, I was able to replicate what you were seeing. Turns out that on smaller workspace when a user joins the workspace, we automatically open DMs with between that user and about 10 other users. So it's possible that the DM channels with your bot already existed because they were created as soon as the user was added to the workspace, hence no im_created events.
I tested some more by inviting new users to my small workspace while listening to the im_created event subscribed at a workspace level (not a bot level). As soon as the new user accessed the workspace what do you know? An im_created event is fired. However, this doesn't work if you subscribe at a bot level."
For this use case (welcoming user when starting conversation with the bot), one should use 'app_home_oppened' event and test if it is the beginning of the conversation with your own backend/data, quoting again the Slack support :
"Any way you cut it, this would not have been the right event for your user case, app_home_opened is the right call."
So having been to Ignite in 2017 I was really excited with the new possibilites the Webhooks in Microsoft Graph and binding to Azure functions.
Recently I got the chance to really explore this for myself.
I am looking at this from the perspective of Identity Management - I really want to see what kind of user onboarding/change management we can react to and process with Graph Webhooks and Azure functions. So I started looking at the beta endpoint and the Webhooks available for "/users"
The first thing that struck me was that in the beta only "updated" or "deleted" is a valid changetype. I really would like "created" - since that is when the most work would get done on a user (for example generating some unique attribute values)
Ok I thought and just tried looking at "/users" and only changetype="updated". I created a subscription and an basic Azure function to handle the requests. Updated a user in Azure AD (just changed "Last name" attribute) and sure enough a trigger was sent to my Azure function
Now comes my biggest problems - this is really unusable in its current form.
his seems to react to the all changes /users and i guess the trigger response could contain several users.
It really would be preferable to get individual triggers for each object changed in /users even though they were changed at the same time
Looking at the actual information sent here lies the BIG problem.
I get the id of the user changed (good, but also expected)
I get the organizationId (ok..)
I get the eventTime (good)
I get a sequenceNumber (unsure what this is?)
I get subscriptionExpirationDateTime (ok fine good to have)
I get subscriptionId for the webhook (ok fine good to have)
... but WHERE is information about what data was changed??? Nowhere to be found is what attributes of the user were changed (i my case "Last name"). This makes the triggers totally unusable for "/users" and I cant really think of anyone who could use this function as is?
Sure I know the object was changed but I have no idea WHAT happened and if the change was relevant to my function
Please tell me there are plans to include the actual changes in the trigger response?
EDIT: ok right, yeah this is more a feature request from the actual developers of ms graph - will look for a better place to get this answered
Please provide feature requests here (ex: richer data in notifications, "created" change type) : https://officespdev.uservoice.com/forums/224641-feature-requests-and-feedback?category_id=101632
Here are answers to other questions.
Microsoft graph doesn't guarantee ordering of events when sending notifications (ex: your webhook endpoint could be down and we will retry events delayed by up to 4 hours or drop if the outage is longer than 4 hours). Hence "SequenceNumber" can be used to track if an event is in order and hence used as is or if it is out of order and needs a query to Graph to get current state.
Currently, we provide Ids of objects and associations (member, manager) that have changed, whether the object/associations is deleted or updated but not details of other properties that were changed. In its current form, webhook is best used with delta query. Instead of polling delta query every X minutes and in most cases receiving zero changes, developers can create a subscription and perform delta query only when a notification is received. This would help scale in case there are many tenants that needs to be polled.
Delta Query: https://developer.microsoft.com/en-us/graph/docs/concepts/delta_query_overview
Also FYI, webhook notifications for user/group is now also available in V1.0
The CloudKit WWDC videos recommend implementing sync like this:
Track local changes
Send changes to the server
Resolve conflicts
Fetch server changes with CKFetchRecordChangesOperation
Apply server changes
Save server change token
I'm following this pattern in my app, but I'm running into a problem with deletion and parent-child relationships.
Let's say we have a list of books that are split up into categories. Every book has to belong to exactly one category.
I start with data like this:
SERVER
Thrillers: "Look Out!", "Secret Spy"
Non-Fiction: "Sailing the Seas", "Gardening Adventures"
Computer Programming: <empty>
As you can see, the final category is empty. Let's say I have two devices with exact copies of this data.
Now, on Device 1, the user adds a book CloudKit Sync to "Computer Programming":
DEVICE 1
Thrillers: "Look Out!", "Secret Spy"
Non-Fiction: "Sailing the Seas", "Gardening Adventures"
Computer Programming: "CloudKit Sync"
But on Device 2, the user completely deletes the "Computer Programming" category (it's empty, so this is fine from Device 2's point-of-view):
DEVICE 2
Thrillers: "Look Out!", "Secret Spy"
Non-Fiction: "Sailing the Seas", "Gardening Adventures"
Device 1 syncs first, so it creates a new Book entry with its parent field set to Computer Programming.
But now Device 2 starts its sync process. It applies its changes to the server, so it deletes the CKRecord corresponding to "Computer Programming". This is consistent with Device 2's worldview, where the category is empty and can be deleted.
However, when it deletes this category from the server, this doesn't make sense with respect to the worldview of Device 1 and the server itself. There's now an orphan book called CloudKit Sync that has a dangling pointer to its parent.
If I'm following Apple's recommendations from WWDC, how do I avoid this scenario? Depending on the order of the sync, I can easily arrive at an inconsistent state with an orphaned book and an invalid parent reference.
What I'd like to happen is for the Delete command from Device 2 to return an error telling me I'm going to orphan a book and prevent the action from occurring at all, so I can take some action to fix the situation.
Is that possible? Is there another way to approach this?
Yes, the behavior you want for Device 2 is possible. I see three aspects of cloudkit that will come into play in your scenario. Let's look at those first, then how they might be used in your scenario.
First, assuming that both (or all) devices have subscribed to changes to the appropriate records, each device would be notified that someone else added or removed something. The device receiving the alert would then have the opportunity to decide what to do about it. (remove it from it's local view, replace it on the server, etc)
Second, you can set the behavior for handling conflicts using the savePolicy on the CKModifyRecordOperation. You can specify whether the last change should overwrite older records, throw an error, etc. See https://developer.apple.com/documentation/cloudkit/ckrecordsavepolicy?language=objc for the three options. (I've only used this in the context of two users modifying a common record, but a deletion after another user updated the record should then throw a server record changed error).
Third, assuming you've configured the aforementioned savePolicy, is the server change token itself. I find it easiest to envision the change token as just a last-modified timestamp. "My copy of this record was last modified at 10:42pm" kind of thing. Depending on the overwrite options you've selected in the aforementioned savePolicy, the device will receive an NSError Server Record Changed alerting you that the version on the server is from, say, 10:56pm, and that your local version may no longer be valid.
The userInfo in the resulting NSError includes 3 versions of the record in question: the current version on the server, the version you tried to submit, and the common ancestor version. The guides from Apple say it's up to the developer to decide what how to merge this information. But in theory, you'd be able to diff the changes, decide which you want to keep, and then submit a new operation.
Regarding your specific scenario: Assuming you fully authorize and trust both dev1 and dev2 to delete records, then I would subscribe to creation and deletion events, and set the savePolicy to throw an error when attempting a conflicting change. In this case, Device 1 would add the record and Device 2 would receive the notification of the new record. If Device 2 simply attempts to delete the old record, it should fail with a server record changed error, which you could display to the user as
"Someone else modified this record, do you really want to delete it
(y/n)."
Device 2 would have to refresh the record (and receive the new record change token) before proceeding. After that, if Device 2 still wants to delete the new record, it could, but then Device 1 would be notified of the change via the aforementioned subscription. Device 1 would then download the new record to (or in this case remove the old record from) its local view. The subscription notification could alert user 1:
"Your record Foo was just deleted by Bar"
This will work even if the events happen practically simultaneously, because one of the changes will be applied on the server first and the other device's token will immediately become out-of-date. So, if Device 2 managed to delete the record first, Device 1's attempt to modify the record will fail with server record changed because Device 1's change token is now out of date. Device 1's error handler would have to decide whether to honor the deletion or to proceed with creating a new record based on your business rules. Maybe ask user 1 with something like:
"Computer Programming" has been removed from the server. Do you want to recreate
it?
At this point, user1 can send flame emails demanding other users stop deleting their newly created records, and user2 can demand that people stop recreating the records they just "cleaned up." :)
You could get a lot more complicated, maybe giving device 1 precedence over device 2, such that when device 1 is notified that the record is deleted, then device 1 re-writes the record to the server. If you have multiple users with deletion rights, you could determine an order of precedence and build out the appropriate error/notification handlers. However, this seems excruciating complicated and error prone. Loops that auto respond (create, delete, create, delete, create, delete) could occur. I include it only as a hypothetical example, not a recommendation!
Lastly, as a different example, my app has a different scenario. The records in my case are gaming sessions. All players need read access to the session data, but only the originator is given the option to delete the record altogether. So, you might consider whether you really authorize multiple users to delete shared records or not.
I am using the Twitter public stream API to search for some keywords. I am writing my script in Java and therefore I use twitter4j. Now I stumbled over the information about status deletion notices:
Status deletion notices (delete)
These messages indicate that a given Tweet has been deleted. Client
code must honor these messages by clearing the referenced Tweet from
memory and any storage or archive, even in the rare case where a
deletion message arrives earlier in the stream that the Tweet it
references.
https://dev.twitter.com/docs/streaming-apis/messages#Status_deletion_notices_delete
So I created methods to remove records from my database when such a notice occurs. Unfortunately such a notice never occurs. I searched to find out what I am doing wrong and found some posts in the twitter developer section concerning the same problem:
https://dev.twitter.com/discussions/17393
https://dev.twitter.com/discussions/19943
https://dev.twitter.com/issues/1355
https://dev.twitter.com/discussions/12836
but unfortunately all these discussions got no answer. So for me it seems like I did no mistake with my code but twitter4j never sends me an deletion notice.
I want to respect the privacy of the twitter users - at least for legal reasons. So my question is:
What can I do to respect the privacy of the users ?
What do I have to do to satisfy my legal duties ?
One alternative seems to be to periodically iterate through all saved Tweets in my Database and request them from twitter to see whether I get a result back or not (so they were deleted). But this doesn't seem to be a practicable way because the data will get more and more and therefore at some point of time I will have limitations (in time, allowed twitter requests, ...). So what should I do?
Thanks in advance! Your help is greatly appreciated.
Ludwig
twitter4j v.3.0.6
Given the nature of the volume of tweets, it's unreasonable to assume that you would check to see if all the tweets are still there. You should make sure that you properly act on a delete notice from twitter. The onus is on them to actually send the delete notification.
That being said, I receive delete notifications from twitter. However, we aren't using the public stream, we are using sitestreams, which relies on authorizing specific social accounts and streaming all updates for those accounts (e.g. favorites, follows, blocks, tweets, retweets, etc) to us in realtime.
If you are doing a stream with filters, for example, it's probably not feasible (or at least very taxing) to run all deleted items through the same pipeline as new items. Or perhaps, to guess at which you were sent based on the times that you were running your filter.
As noted in the issue you linked to, the public streaming API will not necessarily send them out. I'd endeavor to handle them, and possibly provide a tool to manually remove any if a request comes in through another channel, but not worry too much about it, given that twitter doesn't provide the proper facility to be notified of such instances.
I am working on PHP project that should fetch emails from IMAP server, and store them in local database.
Same IMAP server can be used by other email clients, like outbox and so on.
The problem is how to know which messages I already fetched, and which I didn't? I am thinking to use search by datetime, but is it reliable(I would have cronjob, that would access user mail box every minute, and check for emails, but not sure if datetime can cause some issues, for example in case when at almost same time arrive short message and message with big attachment).
I was thinking about system tags, but user can modify them via email client, so I can rely on them, and don't want to modify them and confuse client.
Next I was thinking about custom tags, but not all IMAP servers support them(and our software need to be flexible as much as possible).
Any good idea how I could solve this problem?
Keep track of the currently highest synced UID of the folder you are syncing, and verify that the UIDVALIDITY value of the folder match.
Unique identifiers are assigned in a strictly ascending fashion in the mailbox; as each message is added to the mailbox it is assigned a higher UID than the message(s) which were added previously. Unlike message sequence numbers, unique identifiers are not necessarily contiguous.