I am developing a auto-sync application with offline functionality. User will be creating folder and documents and can sync with the server. There is a functionality of auto-sync in which sync with the server will happen in background every hour. During the sync process, user can also create, delete folders/document. All the changes are saved in core data. For changes to be send to server, i am maintaining a bool. If bool is NO then send those items to server and once response received set the bool to YES.
Now my problem is how can i make sure that changes done while sync is in process to be sent to server. Since sync is in process and during this if i make any changes to item which has been sent to server, even though sync bool is set to NO changes from server is update to YES again and these changes will not be send to server.
Note: I doesn't want to make any restrict to user while sync is in process.
How can i achieve this?
I'll tell you how we specifically solved this. In our syncing code, any attribute on any given object which a user can change also has an associated 'dirty flag', simply an additional boolean which is set whenever the user changes a property on an object in the main thread context (name has nameDirty, etc.). This boolean is automatically set based on a context pre-save notification observer, which checks the change dictionary of the context and marks attributes dirty as appropriate.
Sounds similar to your sync flag so far, though yours is per-object rather than per-attribute.
Any changes from the server will NOT be parsed into any attribute which has the dirty flag set to YES. Similarly, whenever a change is synced to the server, that flag is reset to NO so it will accept changes from the server again.
What happens when the user makes a change on the main context and the server parses in a change on the sync context (on a background thread) at the same time? That's where using the correct NSMergePolicy comes in. The main context has a merge policy telling its in-memory changes to trump anything in the database. Similarly, the sync context has a merge policy telling it to let any values in the database trump any in-memory changes in the sync context. (The same pre-save hook also pokes the sync context and tells it to sync any new changes to the server in the background.) This ensures the syncing process can never step on any changes the user just made on the main thread, even in a SQLite-level merge conflict.
The key here is the context separation between the context which takes user changes and the context which parses in server changes, as that allows us to very cleanly prioritize things and ensure bad merges can never happen. Both contexts are tied directly to the persistent store coordinator.
I'm working on a blog post about how our syncing framework works, but it's not quite done, so I don't have a link yet.
I know there are various third-party syncing libraries out there, and due to our unique requirements we chose to write our own, but I would definitely suggest seeing if any fit your needs out of the box.
Related
The CloudKit WWDC videos recommend implementing sync like this:
Track local changes
Send changes to the server
Resolve conflicts
Fetch server changes with CKFetchRecordChangesOperation
Apply server changes
Save server change token
I'm following this pattern in my app, but I'm running into a problem with deletion and parent-child relationships.
Let's say we have a list of books that are split up into categories. Every book has to belong to exactly one category.
I start with data like this:
SERVER
Thrillers: "Look Out!", "Secret Spy"
Non-Fiction: "Sailing the Seas", "Gardening Adventures"
Computer Programming: <empty>
As you can see, the final category is empty. Let's say I have two devices with exact copies of this data.
Now, on Device 1, the user adds a book CloudKit Sync to "Computer Programming":
DEVICE 1
Thrillers: "Look Out!", "Secret Spy"
Non-Fiction: "Sailing the Seas", "Gardening Adventures"
Computer Programming: "CloudKit Sync"
But on Device 2, the user completely deletes the "Computer Programming" category (it's empty, so this is fine from Device 2's point-of-view):
DEVICE 2
Thrillers: "Look Out!", "Secret Spy"
Non-Fiction: "Sailing the Seas", "Gardening Adventures"
Device 1 syncs first, so it creates a new Book entry with its parent field set to Computer Programming.
But now Device 2 starts its sync process. It applies its changes to the server, so it deletes the CKRecord corresponding to "Computer Programming". This is consistent with Device 2's worldview, where the category is empty and can be deleted.
However, when it deletes this category from the server, this doesn't make sense with respect to the worldview of Device 1 and the server itself. There's now an orphan book called CloudKit Sync that has a dangling pointer to its parent.
If I'm following Apple's recommendations from WWDC, how do I avoid this scenario? Depending on the order of the sync, I can easily arrive at an inconsistent state with an orphaned book and an invalid parent reference.
What I'd like to happen is for the Delete command from Device 2 to return an error telling me I'm going to orphan a book and prevent the action from occurring at all, so I can take some action to fix the situation.
Is that possible? Is there another way to approach this?
Yes, the behavior you want for Device 2 is possible. I see three aspects of cloudkit that will come into play in your scenario. Let's look at those first, then how they might be used in your scenario.
First, assuming that both (or all) devices have subscribed to changes to the appropriate records, each device would be notified that someone else added or removed something. The device receiving the alert would then have the opportunity to decide what to do about it. (remove it from it's local view, replace it on the server, etc)
Second, you can set the behavior for handling conflicts using the savePolicy on the CKModifyRecordOperation. You can specify whether the last change should overwrite older records, throw an error, etc. See https://developer.apple.com/documentation/cloudkit/ckrecordsavepolicy?language=objc for the three options. (I've only used this in the context of two users modifying a common record, but a deletion after another user updated the record should then throw a server record changed error).
Third, assuming you've configured the aforementioned savePolicy, is the server change token itself. I find it easiest to envision the change token as just a last-modified timestamp. "My copy of this record was last modified at 10:42pm" kind of thing. Depending on the overwrite options you've selected in the aforementioned savePolicy, the device will receive an NSError Server Record Changed alerting you that the version on the server is from, say, 10:56pm, and that your local version may no longer be valid.
The userInfo in the resulting NSError includes 3 versions of the record in question: the current version on the server, the version you tried to submit, and the common ancestor version. The guides from Apple say it's up to the developer to decide what how to merge this information. But in theory, you'd be able to diff the changes, decide which you want to keep, and then submit a new operation.
Regarding your specific scenario: Assuming you fully authorize and trust both dev1 and dev2 to delete records, then I would subscribe to creation and deletion events, and set the savePolicy to throw an error when attempting a conflicting change. In this case, Device 1 would add the record and Device 2 would receive the notification of the new record. If Device 2 simply attempts to delete the old record, it should fail with a server record changed error, which you could display to the user as
"Someone else modified this record, do you really want to delete it
(y/n)."
Device 2 would have to refresh the record (and receive the new record change token) before proceeding. After that, if Device 2 still wants to delete the new record, it could, but then Device 1 would be notified of the change via the aforementioned subscription. Device 1 would then download the new record to (or in this case remove the old record from) its local view. The subscription notification could alert user 1:
"Your record Foo was just deleted by Bar"
This will work even if the events happen practically simultaneously, because one of the changes will be applied on the server first and the other device's token will immediately become out-of-date. So, if Device 2 managed to delete the record first, Device 1's attempt to modify the record will fail with server record changed because Device 1's change token is now out of date. Device 1's error handler would have to decide whether to honor the deletion or to proceed with creating a new record based on your business rules. Maybe ask user 1 with something like:
"Computer Programming" has been removed from the server. Do you want to recreate
it?
At this point, user1 can send flame emails demanding other users stop deleting their newly created records, and user2 can demand that people stop recreating the records they just "cleaned up." :)
You could get a lot more complicated, maybe giving device 1 precedence over device 2, such that when device 1 is notified that the record is deleted, then device 1 re-writes the record to the server. If you have multiple users with deletion rights, you could determine an order of precedence and build out the appropriate error/notification handlers. However, this seems excruciating complicated and error prone. Loops that auto respond (create, delete, create, delete, create, delete) could occur. I include it only as a hypothetical example, not a recommendation!
Lastly, as a different example, my app has a different scenario. The records in my case are gaming sessions. All players need read access to the session data, but only the originator is given the option to delete the record altogether. So, you might consider whether you really authorize multiple users to delete shared records or not.
I want to ask for a safe way to clear subEntities in coredata.
I have my a many-to-many relationship like this: Product *<->* Product. Therefore, I've got to create a subEntity to hold some special values between (sortPosition, groupName.....).
So it's like this: Product *<->1 ProductSubEntity 1<->*Product.
When I download products from server's API, the easiest way to update correctly correspond to the server's result is:
Remove all child relationship ([self removeProductSubEntities:self.subEntities]).
Add sub from server's result.
Result: There'd be a lot of subEntity in coredata (which won't hold relationship to any product), and this might take storage/memory/cpu when CRUD (I think?). But I can't actual delete the subEntity (in case it's being hold reference to as an viewController's Object somewhere, and it might cause crash: access to a deleted object).
QUESTION:
How can I clear those sub entities (might occur sometimes) if:
No relationship to any product.
No actual reference from anywhere (any viewControllers or objects)???
P/S: I'm thinking of implement a batch delete when terminate app. Could that be consider a safe solution?
I don't consider this to be a datastore issue, rather a UI update issue. You should delete the objects from the datastore when you don't need them any more and you should update the UI accordingly.
1 thing you didn't mention is re-use. It's possible that your download may be an update to an existing item, which you could find and update, then life is easy all round. Arguably everything below still applies in this case though as your UI might not update to reflect changes and you may need to refresh the managed object.
For the UI update it's generally wise to observe the datastore for changes, usually with an NSFetchedResultsController. If you're doing this then your UI would automatically update itself with the changes.
If you're explicitly passing entity instances around then you should have some way to trigger an update explicitly, and exactly how that works depends on your UI. Generally speaking you'd be doing something like posting a UINotification to tell the system that the datastore changed and they need to re-validate their data objects. For the UI you shouldn't be showing now-dead objects to the user, and in your question where you talk about not deleting to avoid crashes, it's probably worse to allow the user to update invalid objects and just quietly not telling them that their updates won't be saved. When the notification is received you may want to pop a (some) controller(s) off the stack, or re-query the datastore for the new data to be displayed.
If for some reason you don't want to do the above, then yes, you can query for all of the entities with a nil relationship and then batch delete them. This should be done on a background thread just like data loading and I'd recommend doing it on app load instead of close (because you won't have so many view controllers loaded and the ones that are should all have only valid references now...).
I have already read Rails - How do I temporarily store a rails model instance? and similar questions but I cannot find a successful answer.
Imagine I have the model Customer, which may contain a huge amount of information attached (simple attributes, data in other tables through has_many relation, etc...). I want the application's user to access all data in a single page with a single Save button on it. As the user makes changes in the data (i.e. he changes simple attributes, adds or deletes has_many items,...) I want the application to update the model, but without committing changes to the database. Only when the user clicks on Save, the model must be committed.
For achieving this I need the model to be kept by Rails between HTTP requests. Furthermore, two different users may be changing the model's data at the same time, so these temporary instances should be bound to the Rails session.
Is there any way to achieve this? Is it actually a good idea? And, if not, how can one design a web application in which changes in a model cannot be retained in the browser but in the server until the user wants to commit them?
EDIT
Based on user smallbutton.com's proposal, I wonder if serializing the model instance to a temporary file (whose path would be stored in the session hash), and then reloading it each time a new request arrives, would do the trick. Would it work in all cases? Is there any piece of information that would be lost during serialization/deserialization?
As HTTP requests are stateless you need some kind of storeage between requests. The session is the easiest way to store data between requests. As for you the session will not be enough because you need it to be accessed by multiple users.
I see two ways to achive your goal:
1) Get some fast external data storage like a key-value server (redis, or anything you prefer http://nosql-database.org/) where you put your objects via serializing/deserializing (eg. JSON).
This may be fast depending on your design choices and data model but this is the harder approach.
2) Just store your Objects in the DB as you would regularly do and get them versioned: (https://github.com/airblade/paper_trail). Then you can just store a timestamp when people hit the save-button and you can always go back to this state. This would be the easier approach i guess but may be a bit slower depending on the size of your data model changes ( but I think it'll do )
EDIT: If you need real-time collaboration between users you should probably have a look at something like Firebase
EDIT2: Anwer to your second question, whether you can put the data into a file:
Sure you can do that. But you would need some kind of locking to prevent data loss if more than one person is editing. You will need that aswell if you go for 1) but tools like redis already include locks to achive your goal (eg. redis-semaphore). Depending on your data you may need to build some logic for merging different changes of different users.
3) Another aproach that came to my mind would be doing all editing with Javascript and save it in one db-transaction. This would go well with synchronization tools like firebase (or your own synchronization via Rails streaming API)
I'm just starting to learn ASP.NET MVC and I'd like to know how I can retain model objects between subsequent requests to controller action methods?
For example say I'm creating a contact list web app. Users can create, update, rename, and delete contacts in their list. However, I also want users to be able to upload a contact list exported from other programs. Yet I don't want to just automatically add all the contacts in the uploaded file I want to give the user a secondary form where they can pick which uploaded contacts should be actualy added to their list.
So first I have a ContactController.Upload() method which shows an upload form. This submits to ContactController.Upload(HttpPostedFileBase file) which reads the file that was posted into a set of Contact model objects. Then I want to display a list of all the names of the contacts in the list and allow the user to select those that should be added to their contact list. This might be a long list that needs to be split up into multiple pages, and I might also want to allow the user to edit the details of the contacts before they are actually added to their contact list.
Where should I save the model objects between when a user uploads a file and when they finally submit the specific contacts they want? I'd rather not immediately load all the uploaded contacts into the back end database, as the user may end up only selecting a handful to actually add. Then the rest would need to be deleted. Also I would have to account for the case when a user uploads a file, but never actually completes the upload.
From what I understand an instance of a controller only lasts for one request. So should I create a static property on my Contact controller that contains all the latest uploaded contact model object collections? And then have some process that periodically checks the age of these collections and clears out any that are older then some specified expiration time?
A static property on the controller is trouble. First off, it won't work in a web farm and second it you'd have to deal with multiple requests from different users. If you really don't want to use your database you could use the ASP.NET Session.
No, you don't want a static property, as that would be static to all instances of the controller, even for other users.
Instead, you should create a table used to upload the data to. This table would be used as an intermediary between when the user uploads the data, and completes the process. Upon completion, you copy the contacts you want to keep into your permanent table, then delete the temporary data. You can then run a process every so often that purges incomplete data that is older than a specified time limit.
You could also use the HttpContext.Cache, which supports expiration (and sliding expiration) out-of-the box.
Alternatively, and perhaps even better (but more work) you could use cookies and have the user modify the data using javascript in her browser before finally posting it to you.
However, I'd strongly recommend to store the uploaded information in the database instead.
As you pointed out, it might be a lot of data and the user might want to edit it before clicking 'confirm'. What happens if the user's machine (or browser) crashes or she has to leave urgently?
Depending on the way you store the data the data in this scenario will probably be lost. Even if you used the user id as a cache key, a server restart, cache expiration or cache overflow would cause data loss.
The best solution is probably a combination of database and cookie storage where the DB keeps the information in a temporary collection. Every n minutes, or upon pagination, the modified data is sent to the server and updated in the DB.
The problem with storing the data in session or memory is what happens if the user uploads 50k contacts or more. You then have a very large data set in memory to deal with which depending on your platform may effect application performance.
If this is never going to be an issue and the size of the imported contacts list is manageable you can use either the session or cache to store the dataset for further modifications. Just remember to clear it when the user has committed the changes, you don't want a few heavy datasets hanging around in session.
If you store the dataset in session using your application controller then it will be available to all controllers while it is needed.
I'm developing an iPhone app that uses a user account and a web API to get results (json) from a website. The results are a list of user's events.
Just looking for some advice or strategies - when to cache and when to make an api call... and if the iPhone SDK has anything built in to handle these scenarios.
When I get the results from the server, they populate an array in a controller. In the UI, you can go from a table listing view, to a view of an individual event result - so two controllers share a reference to the same event object.
What gets tricky is that a user can change the details of an event. In this case I make a copy of the local Event object for the user's changes, in case they make an error. If the api call successfully goes through and updates that event on the server, I take these local changes from the Event copy and set the original Event object to match with setters.
I have the original controller observing if any change is made to the local Event object so that it can reflect it in the UI.
Is this the right way of doing things? I don't want to make too many API calls to reload data from the server, But after a user makes an update should I be pulling down the list again with the API call?
...I want to be careful that my local objects don't become out of sync with the remote.
Any advice is appreciated.
I took a similar approach with an app I built. I simply made a duplicate version of the remote data model with Core Data, and I use etags on the backend to prevent sync issues (in my case, it's okay to create duplicate records).
It sounds like you're taking a good approach to this.
Some time back, I developed an iOS app where in, I had almost same requirement to store data on server as well as locally to avoid several network call and also user can see their information without any delay.
In that app, user can store photos, nodes, checkIns and social media post and with all this data, app can form a beautiful timeline. So what we did was, we had everything locally and whenever user phone come in some WIFI zone, we start uploading that data to server and sync both (local and remote) databases.
Note this method works well when only one user can access this data.