I'm developing an app with Core Data that periodically downloads all the data from a webservice. Since the download can fail or be cancelled by the user, I want to be able to roll back to the previous state. I tried undoing the NSManagedObjectContext, but that seemed a bit slow (I have tens of thousands of entities). What I'm doing right now is making a backup of the persistent store file, download the data, and, if the download fails, replace the store file with the backup. This seems to work correctly, except there seems to be a delay after I can fetch entities from the store: if after the download I go immediately to a UITableView that uses an NSFetchedResultsController, I find it empty. If I wait some seconds, everything is ok.
So my question is: has anyone had this kind of delays too? Is there something that can be done to avoid this problem, something that forces everything to be ready, even if it blocks the thread?
I haven't used this setup but I think the delay you are seeing is probably caused by Core Data having to clear all it's caching. Core Data uses If you use a cache with the fetched results controller it will have to test and then delete it's existing cache.
I think the best thing to do is to tear down you Core Data stack and reboot it from scratch. That includes recreating a fresh fetched results controller.
Related
Background story
I am developing a big iOS app. This app works under specific assumptions. The main of them is that app should work offline with internal storage which is a snapshot of last synchronized state of data saved on server. I decided to use CoreData to handle this storage. Every time app launches I check if WiFi connection is enabled and then try to synchronize storage with server. The synchronization can take about 3 minutes because of size of data.
The synchronization process consists of several stages and in each of them I:
fetch some data from the server (XML)
deserialize it
save it in Core Data
Problem
Synchronization process can be interrupted for several reasons (internet connection, server down, user leaving application, etc). This may cause data to be out-of-sync.
Let's assume that synchronization process has 5 stages and it breaks after third. It results in 3/5 of data being updated in internal storage and the rest being out of sync. I can't allow it because data are strongly connected to each other (business logic).
Goal
I don't know if it is possible but I'm thinking about implementing one solution. On start of synchronization process I would like to create snapshot (some kind of copy) of current state of Core Date and during synchronization process work on it. When synchronization process completes with success then this snapshot could overwrite current CoreData state. When synchronization interrupts then snapshot can be simply aborted. My internal storage will be secured.
Questions
How to create CoreData snapshot?
How to work with CoreData snapshot?
How to overwrite CoreDate state with snapshot?
Thanks in advice for any help. Code examples, if it is possible, will be appreciated.
EDIT 1
The size of data is too big to handle it with multiple CoreData's contexts. During synchronization I am saving current context multiple times to cleanup memory. If I do not do it, the application will crash with memory error.
I think it should be resolved with multiple NSPersistentStoreCoordinators using for example this method: link. Unfortunately, I don't know how to implement this.
You should do exactly what you said. Just create class (lets call it SyncBuffer) with methods "load", "sync" and "save".
The "load" method should read all entities from CoreData and store it in class variables.
The "sync" method should make all the synchronisation using class variables.
Finally the "save" method should save all values from class variables to CoreData - here you can even remove all data from CoreData and save brand new values from SyncBuffer.
A CoreData stack is composed at its core by three components: A context (NSManagedObjectContext) a model (NSManagedObjectModel) and the store coordinator (NSPersistentStoreCoordinator/NSPersistentStore).
What you want is to have two different contexts, that shares the same model but use two different stores. The store itself will be of the same type (i.e. an SQLite db) but use a different source file.
At this page you can see some documentation about the stack:
https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/CoreData/InitializingtheCoreDataStack.html#//apple_ref/doc/uid/TP40001075-CH4-SW1
The NSPersistentContainer is a convenience class to initialise the CoreData stack.
Take the example of the initialisation of a NSPersistentContainer from the link: you can have the exact same code to initialise it, twice, but with the only difference that the two NSPersistentContainer use a different .sqlite file: i.e. you can have two properties in your app delegate called managedObjectContextForUI and managedObjectContextForSyncing that loads different .sqlite files. Then in your program you can use the context from one store to get current data to show to the user and you can use the context that use the other store with a different .sqlite if you are doing sync operations. When the sync operations are finally done you can eventually swap the two files and after clearing and reloading the NSPersistentContainer (this might be tricky, because you will want to invalidate and reload all managed objects: you are switching to an entirely new context) you can then show the newly synced data to the user and start syncing again on a new .sqlite file.
The way I understand the problem is that you wish to be able download a large "object graph". It is however so large that it cannot be loaded at once in memory, so you would have to break it in chunks and then merge it locally into to Core data.
If that is the case, I think that's not trivial. I am not sure I can think of direct solution without understanding the object relations and even then it may be really overwhelming.
An overly simplistic solution may be to generate the sqlite file on the backend then download it in chunks; it seems ugly, but it serves to separate the business logic from the sync, i.e. the sqlite file becomes the transport layer. So, I think the essence to the solution would be to find a way to physically represent the data you are syncing in a format that allows for splitting it in chunks and that can afterwards be merged into a sqlite file (if you insist on using Core data).
Please also note that as far as I know Amazon (https://aws.amazon.com/appsync/) and Realm (https://realm.io/blog/introducing-realm-mobile-platform/) provide background sync of you local database, but those are paid services and you would have to be careful not be locked in (should not depend on their libs in your model layer, instead have a translation layer).
My question surrounds around one single point - data management in mobile application. I have created a mobile application where data comes from server. The data includes both text and images. Following are the steps I am doing for this :
First launch :
1. Get server data.
2. Save server data in Sqlite database.
3. Show Sqlite data.
Next launches :
1. Show Sqlite data.
2. Get server data in background.
3. Delete previous Sqlite data.
4. Save new server data in Sqlite database.
5. Show Sqlite data.
I have couple of questions on these steps :
1. Is this the right approach ? Other way could be showing data every time from server but that would not display the data on screen immediately (depending on internet speed).
2. I also thought of comparing the Sqlite data with the new server data. But faced a big challenge. The new server data might have new records or deleted records. Also, I could not find an appropriate approach to compare each database field with JSON data.
So what is the best approach to compare local Sqlite data with new server data ?
3. Each time I delete the Sqlite data and insert new data and then refresh the screen (which has a UITableView), it blinks for a second which is obvious. How to avoid this issue if steps 3, 4, 5 are followed ?
4. How should I proceed with data update in case I come back on the screen each time or when the application becomes active ? I am very aware of NSOperationQueues or using GCD for that matter. But what if I am crazy and go back and forth to screen again and again. There will be a number of NSOperations in the queue.
It's a challenge to synchronise server data, I've done that before, and if you can spend time on it I'd say it's the best solution.
You may need creation and modification dates on both server and local objects, to compare them - this will let you decide which objects to add, update and delete.
If the server sends you only the recently updated objects you can save a lot of traffic and improve performance (but deleted objects will be harder to detect).
If the data is only changed in the server it's easier, when the app can change the data too it becomes more complicated (but it seems that it's not your case). It also depends on how complex the database is, of course.
If you don't want to invest some time in doing this, just fetching all data everytime works too, even if it is not ideal! Instead of showing the old data and blinking it, you can just make the user wait 2-3 seconds when entering, while you get the new data. Or instead you can fetch the data only when starting the app, and so when you get to that view controller it will be ready already.
It's a complex problem that everyone faces at some point, so I'm curious to see what other people will suggest :)
This is a good question.
I personally think downloading data, store locally and later try to sync is a dangerous scenario. Easy to introduce bugs, master <-> slave issues (what data should be master, if multiple devices would be used etc.)
I think something like this could be a working approach:
1. I would try to look at possibilities to lazy load the data from the server on-demand. That is when a user have a View that should display data, load that specific data with the creation of that specific View. This ensures the data is allways in sync.
2. Tackling the need to reload data from server from every view, could be done by simply storing the downloaded data as objects in memory (not using SqlLite). The view will try to load the needed data trough your cache manager, and it would serve it from memory, if available. If not in memory simply get the data from your server and add it to your memory cache.
The memory cache could be a home made data manager wrapping a Dictionary stored on you AppDelegate, or some global "Singelton" to wrap the cache management/storing and data loading.
3. With lazy loaded data and memory cache you would need to make sure any updates (changes, new records, deleted records) updates your memory data model, as well as pushing these changes to the server as soon as possible. Depending on data size etc. you could force the user to wait, or do it directly as background process.
4. To ensure the data is in sync, you should make sure that you periodically invalidate (delete) the local memory records in the cache and thereby force data updates from the server. Best approach would probably be to have a last updated timestamp for each record in the memory cache. So the periodical invalidator would only delete "old records" from the memory cache (once again not from the server).
To save server from unnecessary data load, the data should still load on demand when the user needs it in a view, and not as part of "cache invalidation".
5. Depending on the data size you might need to look at "cache invalidation". Could be as simple as when xx records are stored, start deleting old objects from memory cache (not server, only locally on device).
6. If data sync is absolutely critical you might want to look at refreshing your memory cache for a record, just before you allow the user to change data. E.g. when user taps "Edit" or similar, you grab the latest data from server for that record. This is just to make sure the user is not going to update a record using outdated data and thereby accidentally overriding any changes made remote, or on another device etc.
--
My take on it. I do not believe there is a "perfect right way" to do this. But this would be what I would try to do.
Hope this will help with some ideas and inspiration.
How about this:
If data exists in SqlLite, load into "in-memory" copy and show it
In background load new server data
delete old sqlite data if it exists (note that the in-memory copy remains)
save new server data to sqlite
load new sqlite data into "in-memory" copy and show it.
If no data was found in step 1, display a "loading" screen to the user during step 2.
I'm making the assumption that the data from SqlLite is small enough to keep a copy in memory to show in your UITable view (The UITable view would always show data from in-memory).
It may be possible to combine steps 4 and 5 if the data is small enough to hold two copies in memory at the same time (you would create a new in-memory copy and swap with the visible copy when complete).
Note:
I don't talk about error handling here, but I would suggest that you don't delete the sqlite data until you have new data to replace it with.
This approach also eliminates the need to determine if this is the first launch or not. The logic always remains the same which should make it a little easier to implement.
Hope this is useful.
You can do same things more efficiently by MultiVersion Concurrency Control (MVCC), which uses a counter (sort of a very simple "time stamp") for every data record, which is updated whenever the record is changed means you need to get those data which is Updated after last sync call that reduces lots of redundant data and bandwidth.
Source: MultiVersion Concurrency Control
I'm having an issue where my core data does not persist sometimes. This is the scenario I'm running. I run the app when I'm not logged in, login, and the app stores some data. I then run the app again, this time as the logged in user, and pull that data. It was all working well until I tried to run the app as if it had been downloaded for the first time. To do this I deleted the app from my phone and ran the above steps. However now no data is pulled from the Core Data. Here are some interesting notes and things I have done to remedy the situations.
1) I know with 100% certainty that the data persists if the app is not being run as if just downloaded.
2) The data occasionally persists when run as if just downloaded, but the minority of the time.
3) When I run the app as if just downloaded and store the data into the Core Data, I do an immediate check to see if the data was stored and it is. When I run it again, however, the data is gone.
I'm refraining from posting code right now just because it's very large and would take a lot of time to shorten for the sake of posting. However if you think it'll be helpful in light of what I have said I will post it.
You state that you save with [document saveToURL:forSaveOperation].
In order to persist Core Data you have to save the context.
[managedObjectContext save:nil];
Make sure you call this with a valid context before you expect your data to be persisted. I suspect you have this in some places but not in all required places. One way to find out is to put a breakpoint on each save statement and check if the saves are called as expected.
I am creating an app where data needs to be displayed right away from the local datastore. I fire off a background thread when the app starts to determine if iCloud is available.
I looked everywhere and can't find the solution to this: when iCloud becomes available, can I change the "options" on the persistentStore to start using iCloud transactions?
I'm not sure what the proper approach is in this situation. Everything I try causes the application to crash.
Originally I had it so the iCloud checking wasn't in a background thread and the app worked fine, but occasionally timed out.
You have not to know when iCloud becomes available. You just work with data but you don't send them directly to iCloud. iOS does it instead you. So only it knows when and how it should send data.
No, you can't change the options on an NSPersistentStore object once it exists. You can only specify the options when adding the persistent store to the NSPersistentStoreCoordinator. The closest you could get to changing options would be to tear down the entire Core Data stack and start over with different options.
That wouldn't help, though, because:
Even if you have detected that iCloud is available (I'm guessing using NSFileManager, either via its ubiquityIdentityToken or by calling URLForUbiquityContainerIdentifier:), your call to addPersistentStoreWithType:configuration:URL:options:error: might still block for a while. If there's new data available in iCloud, it doesn't start downloading until you add the persistent store, and that method doesn't return until the download process is finished. And sometimes, iCloud just makes that method block for a while for no readily apparent reason.
If you let the user make any changes to the data while using non-iCloud options, those changes will not get automatically sent to the cloud later. Core Data only sends changes to iCloud when the data changes while iCloud is active-- which makes it generate a transaction. You'd have to load and re-save any changes the user made, or those changes would never make it to the cloud.
You have, unfortunately, hit on one of the major stumbling points when using Core Data with iCloud. You can't make the full data store available until Core Data finishes communicating with iCloud-- because your call to add the persistent store doesn't return until then. And you can't do anything to speed up that process. This is just one of the headaches you'll run into if you continue trying to use iCloud with Core Data.
Depending on the nature of your data you might be able to use two data stores, one purely local and one synced via iCloud. You could make the purely local data store available while the iCloud one tries to get its act together well enough to be useful. If you stick with one data store though, you're stuck with the delay.
Current Setup:
MVC-1, onViewDidLoad, creates a Managed Object, fetches data from the Managed Object and updates various UI elements.
MVC-2: Same exact setup. I copied and pasted the same code from MVC-1 into MVC-2.
Issues I am seeing:
After launch and opening MVC-1, all the code is executed without any errors. The ManagedObject is created, the fetch requests on the ManageObjectContext work and the UI is properly updated. However, when switching to MVC-2 it seems that none of the data, that MVC-2 is suppose to fetch, is actually being updated. All the UI Elements in MVC-2 have the same data from the last time the application was launched.
Furthermore, if I launch the application and open MVC-2 first I get the same results, only MVC-1 does not appear to be updated this time around.
What is it that I am over looking? I have explicit saves to the database being made. I am (at least I think I am) creating two different ManagedObjects. Do I need to somehow close one before creating the other? Any advice would be very helpful. Thanks.
Figured it out.
I was saving the MOC, however I was not writing the NSManagedDocument to disk (I am using a ManagedDocument as a container for the Core Data database).
So naturally, when I switched to the MVC-2 I would not have the updated data.
I now save the MOC, write the MD to disk and close the MD at viewWillDisappear.
Thanks for the help gusy.