NSFileCoordinator error when using UIManagedDocument in iOS 5.0 simulator - ios

I am using a UIManagedDocument in iOS 5.0, running the app on the simulator, using XCode 4.2 under OSX 10.6. The code in question looks as follows:
if (![[NSFileManager defaultManager] fileExistsAtPath:[self.photoDatabase.fileURL path]]) {
// does not exist on disk, so create it
[self.photoDatabase saveToURL:self.photoDatabase.fileURL forSaveOperation:UIDocumentSaveForCreating completionHandler:^(BOOL success) {
[self setupFetchedResultsController];
[self fetchFlickrDataIntoDocument:self.photoDatabase];
}];
} else if (self.photoDatabase.documentState == UIDocumentStateClosed) {
// exists on disk, but we need to open it
// *** the following line generates the message ***
[self.photoDatabase openWithCompletionHandler:^(BOOL success) {
//[self setupFetchedResultsController];
}];
} else if (self.photoDatabase.documentState == UIDocumentStateNormal) {
// already open and ready to use
[self setupFetchedResultsController];
}
Running the marked line creates the following message on the log:
2012-01-10 22:33:17.109 Photomania[5149:4803] NSFileCoordinator: A surprising server error was signaled. Details: Connection invalid
After the message is sent, the UIManagedDocument may or may not work—I have not found the circumstances that determine this, yet.
I am pretty sure that the code is correct, as it's actually one of the code examples in the CS193p course from Stanford. The whole example can be downloaded at their website under
http://www.stanford.edu/class/cs193p/cgi-bin/drupal/
Direct link to the code:
http://www.stanford.edu/class/cs193p/cgi-bin/drupal/system/files/sample_code/Photomania_0.zip
Additionally, the code runs fine on the device itself, without generating the "surprising" message, and running all the code that comes afterwards just fine.
I have not found anything on Google, neither on the Apple Developer pages. Restarting the simulator, or XCode, or reinstalling both of them does not change the behaviour.
Any ideas?

I can only say that I've had this happen to me several times. For me, I'm lazy after I update my dataModel and so far, each time I've gotten this error it was because I had changed my data model. Usually, all I need to do is delete my app from the simulator and re-run it and it has always turned out fine. Hope this helps someone out there.

I think I have found the answer. It looks like the automatic saving for UIManagedDocument kicks in only after a few seconds on the simulator.
So I minimized the app on the simulator, by pressing the home button, and then clicked on the icon to maximize it again. And then I terminated the app in simulator.
When I re-launched the app, the database was loaded. The error still shows up - it comes because the document is in "closed" state (that's normal - that's why CS193P asked to call openWithCompletionHandler), but my data across launches is preserved. Unfortunately I have to do the minimize/maximize routine before terminating the app, or the changes are discarded at next launch.
Can you verify that this is the behavior you are able to recreate? At least for testing purposes this should be a good enough trick to use.

Try upgrading to the latest iOS 5.1. I don't think UIManagedDocument with iCloud works reliably in 5.0. This has been my experience.

I love the Stanford iTunes class. However, I think the sample code for using UIManagedDocument is wrong. In fact, he notes in the demo that he is only doing it that way because he wants to just fetch the information right then. In the code comments, he says not to use the auto-save features because the data will not be saved if the app quits. however, UIManagedDocument will save anything that's necessary before quitting. It has all pertinent handlers for quitting/multitasking/etc to make sure the data is saved.
So, if you are using that code as your example, here's a version that should work, and does not use saveToURL (I don't have a flickr account, so I didn't actually run it - but this is how the class is designed to work). Please let me know if it does not work.
- (void)fetchFlickrDataIntoDocument:(UIManagedDocument *)document
{
NSManagedObjectContext *ctx = [[NSManagedObjectContext alloc] initWithConcurrencyType: NSPrivateQueueConcurrencyType];
ctx.parentContext = document.managedObjectContext;
[ctx performBlock:^{
NSArray *photos = [FlickrFetcher recentGeoreferencedPhotos];
for (NSDictionary *flickrInfo in photos) {
[Photo photoWithFlickrInfo:flickrInfo inManagedObjectContext:ctx];
// Push changes to document MOC
[ctx save:0]; // propagates changes to parent MOC
// and tell the document it is dirty and needs to be saved
// It will be saved when the document decides its time to save
// but it *will* be saved.
[document updateChangeCount:UIDocumentChangeDone]
}
}];
}

Still had errors when the last path component for document file url was #"Database". Adding an extension #"Database.db" seems to have fixed it, everything running fine now. Have also upgraded to Lion though.
NSURL *url = [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
url = [url URLByAppendingPathComponent:#"Database.db"];

Related

Can't read or create UIManagedDocuments anymore

I have a big issue in my app, which prevents creating new documents and reading them, whereas it worked well until now.
I didn't change anything, and it started bugging from a build to another.
This is the code I'm using:
CLProject *project = [[CLProject alloc] initWithFileURL:projectURL];
NSLog (#"Will save project at URL: %#", projectURL);
[project saveToURL:projectURL forSaveOperation:UIDocumentSaveForCreating completionHandler:^(BOOL success) {
NSLog (#"Project saved: %d", success);
[...]
}];
CLProject is a subclass of UIManagedDocument.
The first NSLog is called, but not the second one. Instead I get an error :
2018-02-14 19:21:03.597495+0100 CamList[2247:750786] Will save project
at URL:
file:///var/mobile/Containers/Data/Application/151E38F5-2214-4876-A188-2AB8B5E8CF6A/Documents/Projects/715A0087-F2EF-439B-A2DD-8E878EF8A973.camlist
2018-02-14 19:21:03.783397+0100 CamList[2247:750886] [default] [ERROR]
Could not get attribute values for item
/var/mobile/Containers/Data/Application/151E38F5-2214-4876-A188-2AB8B5E8CF6A/Documents/Projects/715A0087-F2EF-439B-A2DD-8E878EF8A973.camlist
(n). Error: Error Domain=NSFileProviderInternalErrorDomain Code=1 "The
reader is not permitted to access the URL."
UserInfo={NSLocalizedDescription=The reader is not permitted to access
the URL.}
But it doesn't crash, the app keeps running (but nothing happens because the completion block never gets called).
What I don't understand is that everything was working fine and I haven't changed anything...
Can you help me??
Thanks
Well, it seems to work fine again this morning... Nothing to understand. My iPhone had to be tired...

Core Data - can't achieve a simple LightWeight Migration

I am unable to achieve a simple lightweight migration by simply adding 1 Entity to the datamodel.
I have read and followed all the guides/documentation/posts/answers, I can't seem to find my mistake/error.
I do have created a new datamodel from the already existing one.
I do have set the new datamodel as current datamodel.
I do have only added 1 Entity to the new datamodel (+ link to the parent Entity).
I do have passed the dictionary options NSMigratePersistentStoresAutomaticallyOption and NSInferMappingModelAutomaticallyOption in the method addPersistentStoreWithType.
I even tried to log everything, thank to the method given from this post: core data migration
/*! The method checks the Core Data file version is compatible with the App's model version
and then pushes the main menu view onto the navigation stack. If not compatible it displays a
message to the user.
#param file The file URL for the Core Data Store. With UIManagedDocument you have to get the
actual store file URL, you can't just use the UIManagedDocument file URL.
*/
-(void) checkCoreDataFileVersion:(NSURL*)file
{
if ([self checkVersion:file]) {
// file version is compatible so continue (add code to push the menu view)
} else {
// file version is NOT compatible
_fileOpenErrorAlert = [[UIAlertView alloc] initWithTitle:#"Unable to open Document" message:#"Please check that you have the correct application version installed" delegate:nil cancelButtonTitle:#"OK" otherButtonTitles:nil];
[_fileOpenErrorAlert show];
}
return;
}
/*! Checks the Core Data files models version against the apps model version to see if they
are compatible. This will return YES if a lightweight migration can be performed and NO if NOT.
#param fileURL The file URL for the Core Data Store. With UIManagedDocument you have to get the
actual store file URL, you can't just use the UIManagedDocument file URL.
#return Returns YES if they are compatible and NO if not.
*/
- (bool)checkVersion:(NSURL*)fileURL {
NSManagedObjectModel *model = [self managedObjectModel];
NSLog(#" app model entity version hashes are %#", [model entityVersionHashesByName]);
NSError *error;
NSDictionary *metaData = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:NSSQLiteStoreType URL:fileURL error:&error];
if (!metaData) {
NSLog(#"problem getting metaData");
NSLog(#" - error is %#, %#", error, error.userInfo);
return NO;
}
bool result = [model isConfiguration:nil compatibleWithStoreMetadata:metaData];
if (!result) {
NSLog(#" file is not compatible!");
NSLog(#" metadata is %#", metaData);
}
return result;
}
When I make a diff of the metadata from all the Entities, I only match difference for 1 Entity (the newly created). So why it can't make a migration ? I just added 1 Entity.
EDIT :
I don't have Crashes, the App is working fine.
There is something I don't understand. When I download our lastest App from the AppStore, launch it and when I build from xCode my lastest developement App (with the new datamodel) over the one from the AppStore, the migration doesn't occur.
BUT when I use GIT, when I put the HEAD to the lastest release TAG, build, launch the App. Then put back the HEAD to my lastest development feature (with the new datamodel etc..), build and run, the migration is done and everything is working.
So which scenario should I trust ?
Yes, You should trust the 2nd senario to test coredata migration by applying it to the last released code.
The first senario is no more valid since Apple for some security reasons nomore give the ability to update an itune-downloaded app using xcode directly.
There was a way to test the upgrade on itune-version but not directly from xcode.
Technical Note TN2285
Testing iOS App Updates
Install an ad hoc distribution of an archived build of the update
using iTunes on a device that already has the old version of the app
installed.
Installing Your App on Test Devices Using iTunes

Getting an audio device with OpenAL

I'm trying to use OpenAL for an IOS game I'm working on, but having an issue opening the audio device. Specifically, when I call the function alcOpenDevice(NULL), I get 'NULL' in return. This is causing issues, of course, but I can't tell what I'm doing wrong.
I'm new to OpenAL, so I've been looking at a couple guides here and here to see what I need to do. If I download their sample projects and test 'em, they both work fine. If I copy their files into my project, and ignore the files I made, they still work fine. I'm assuming something got lost in translation when I started rebuilding the code for use in my project. Asking around and searching online hasn't given me any leads though, so I'm hoping someone here could put me on the right track.
Here's the actual setup code I'm using in my AudioPlayer.m
- (void)setup {
audioSampleBuffers = [NSMutableDictionary new];
audioSampleSources = [NSMutableArray new];
[self setupAudioSession];
[self setupAudioDevice];
[self setupNotifications];
}
- (BOOL)setupAudioSession {
// // This has been depricated.
//
// /* Setup the Audio Session and monitor interruptions */
// AudioSessionInitialize(NULL, NULL, AudioInterruptionListenerCallback, NULL);
//
// /* Set the category for the Audio Session */
// UInt32 session_category = kAudioSessionCategory_MediaPlayback;
// AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, sizeof(session_category), &session_category);
//
// /* Make the Audio Session active */
// AudioSessionSetActive(true);
BOOL success = NO;
NSError *error = nil;
AVAudioSession *session = [AVAudioSession sharedInstance];
success = [session setCategory:AVAudioSessionCategoryPlayback error:&error];
if (!success) {
NSLog(#"%# Error setting category: %#", NSStringFromSelector(_cmd), [error localizedDescription]);
return success;
}
success = [session setActive:YES error:&error];
if (!success) {
NSLog(#"Error activating session: %#", [error localizedDescription]);
}
return success;
}
- (BOOL)setupAudioDevice {
// 'NULL' uses the default device.
openALDevice = alcOpenDevice(NULL); // Returns 'NULL'
ALenum error = alGetError(); // Returns '0'
NSLog(#"%i", error);
if (!openALDevice) {
NSLog(#"Something went wrong setting up the audio device.");
return NO;
}
// Create a context to use with the device, and make it the current context.
openALContext = alcCreateContext(openALDevice, NULL);
alcMakeContextCurrent(openALContext);
[self createAudioSources];
// Setup was successful
return YES;
}
- (void)createAudioSources {
ALuint sourceID;
for (int i = 0; i < kMaxConcurrentSources; i++) {
// Create a single source.
alGenSources(1, &sourceID);
// Add it to the array.
[audioSampleSources addObject:[NSNumber numberWithUnsignedInt:sourceID]];
}
}
Note: I'm running IOS 7.1.1 on a new iPad air, and using Xcode 5.1.1. This issue has been confirmed on the iPad, my simulator, and an iPod touch.
The Short Answer:
Apple's implementation of alcOpenDevice() only returns the device once. Every subsequent call returns NULL. This function can be called by a lot of Apple audio code, so take out EVERY TRACE of audio code before using OpenAL and manually calling that function yourself.
The Long Answer:
I spent half a day dealing with this problem while using ObjectAL, and ended up doing exactly what you did, re-making the entire project. It worked, until out of curiosity I copied the entire project over, then same problem again, alcOpenDevice(NULL) returned NULL. By chance I stumbled upon the answer. It was this bit of code in my swift game scene:
let jumpSound = SKAction.playSoundFileNamed("WhistleJump.mp3", waitForCompletion: false)
And then I remembered I had this problem before without SKAction involved. That time it turned out I was using ObjectAL in two different ways, I used OALSimpleAudio in one place, and OpenAL objects in another, and it was initializing my audio session twice.
The common thread between these two incidents is both times alcOpenDevice() was called more than once during the life of the application. The first time it was ObjectAL calling it twice due to my misuse of the library. The second SKAction.playSoundFileNamed() must have called alcOpenDevice() before my ObjectAL code did. Upon further research I found this bit in the OpenAL 1.1 Specification:
6.1.1. Connecting to a Device
The alcOpenDevice function allows the application (i.e. the client program) to connect to a device (i.e. the server).
ALCdevice * alcOpenDevice (const ALCchar *deviceSpecifier);
If the function returns NULL, then no sound driver/device has been found. The argument is a null terminated string that requests a certain device or device configuration. If NULL is specified, the implementation will provide an implementation specific default.
My hunch is that Apple's implementation of this function only returns the correct device ONCE for the life of the application. Every time alcOpenDevice is called after that, it returns NULL. So bottom line: Take out every trace of audio code before switching to OpenAL. Even code that seems safe, like SKAction.playSoundFileNamed() still might contain a call to alcOpenDevice() buried deep in its implementation.
For those using ObjectAL, here is the console output of this problem to help them find their way here from google, as I couldn't find a good answer myself:
OAL Error: +[ALWrapper openDevice:]: Could not open device (null)
OAL Error: -[ALDevice initWithDeviceSpecifier:]: <ALDevice: 0x17679b20>: Failed to create OpenAL device (null)
OAL Error: +[ALWrapper closeDevice:]: Invalid Enum (error code 0x0000a003)
OAL Warning: -[OALAudioSession onAudioError:]: Received audio error notification, but last reset was 0.012216 seconds ago. Doing nothing.
fatal error: unexpectedly found nil while unwrapping an Optional value
This SO answer seems to validate my comment about AVAudioSession conflicting with OpenAL. Try removing AVAudioSession, or initializing OpenAL first (tho I imagine this would cause the inverse problem).
Alright, so I ended up starting over in a fresh project with a copy-pasted version of AudioSamplePlayer from the first sample project I linked. -It worked.
I then edited it step by step back to the format I had set up in my project. -It still works!
I still don't know what I did wrong the first time, and I'm not sure it was even in my audio player anymore, but It's running now. I blame gremlins.
...maybe alien surveillance.

(IOS) Cordova Camera Plugin Referring to deleted images

I'm building a cordova app (primarily for IOS & Android) in which the user can take an image, retake (, etc.) it and save it locally.
I'm currently struggling with the cordova cameraPlugin. So, here a short description of the problem.
When the user takes an image, it's saved locally in the apps temp folder and the user is able to view in in the UIWebView. On retaking, the image will be deleted from the temp folder and should not be available any longer (in RAM and local FS).
It works as long as the user doesn't retakes the image 2 or more times, if he does instead of the last image the first image will be referenced/rendered in WebView. After reopening the app, the image is displayed correctly.
An Example:
The user takes the first image. ==> cdv_photo_001.png
The second. ==> cdv_photo_002.png and the first one will be deleted (which seems to work correctly)
And the third. ==> cdv_photo_001.png and the second image will be deleted.
The third image will look the same as the deleted first one. This happens with every image after the third one.
It works fine after restarting the app
I've already tried to disable the App-Cache, delete the app cache before updating the image, refreshing the page and looking for answers online.
I'm getting an error when opening the camera UI, but I could not find a solution for it either.
Snapshotting a view that has not been rendered results in an empty snapshot. Ensure your view has been rendered at least once before
snapshotting or snapshot after screen updates.
The code for the camera Call:
function getPhoto() {
navigator.camera.getPicture(getPhotoOnSuccess, getPhotoOnFail, {
quality: 25,
destinationType: Camera.DestinationType.FILE_URL,
correctOrientation: true,
encodingType: Camera.EncodingType.PNG
});
}
In getPhotoOnSuccess I'm basically saving the image path to a db and appending it with jQuery to the view.
And the code to delete the image: (sidenote I`m new to Objective C)
- (void) deleteImageByPath:(NSString *)imagePath withSelector:(SEL)selector{
NSError *error = nil;
NSFileManager *mgr = [NSFileManager defaultManager];
NSString *tempFolder = NSTemporaryDirectory();
if([mgr removeItemAtPath: imagePath error:&error] == NO) {
NSLog(#"File deleted");
}
//The files can be edited as well, so there can be two files in different directories
if(error != nil){
NSString *imgEl = tempFolder;
imgEl = [imgEl stringByAppendingPathComponent:imagePath.lastPathComponent];
if(![mgr removeItemAtPath:imgEl error:&error]){
NSLog(#"Old element couln't be deleted.");
}
}
[self performSelector:selector withObject:error];
}
The file is not in the directory anymore after deleting it, so I guess it works.
An important detail could be, that I wrote my own IOS cordova plugin, because the method for the file system access provided by cordova sucks.
So thats it.
The specific question is: Why and how is this happening and is there a chance to change this behavior? If yes, how should I proceed?
By the way, I`m using cordova 3.1.0 and the build target is IOS 7.
Thanks in advance.
Ok folks, I finally got it.
The whole problem was not related to my code or any of the cordova code.
So why did it happen? ==> I don't exactly know that, for it seems that this bug or whatever you might call it, has occurred to many people.
And they all tried to delete or deactivate the cache as I did, some of their problems are very close to my own but most aren't, so it took a while til I found a solution.
I read this thread and tried to append a timestamp to the image path and it worked!
My conclusion to this it, that there might be a problem with the UIWebView and the cache management.
Or it might proof as a general WebView problem, I will be able to check that in a few days on an Adroid device.

What is the fastest way to load a large CSV file into core data

Conclusion
Problem closed, I think.
Looks like the problem had nothing to do with the methodology, but that the XCode did not clean the project correctly in between builds.
It looks like after all those tests, the sqlite file that was being used was still the very first one that wasn't indexed......
Beware of XCode 4.3.2, I have nothing but problems with Clean not cleaning, or adding files to project not automatically being added to the bundle resources...
Thanks for the different answers..
Update 3
Since I invite anybody to just try the same steps to see if they get the same results, let me detail what I did:
I start with blank project
I defined a datamodel with one Entity, 3 attributes (2 strings, 1 float)
The first string is indexed
In did finishLaunchingWithOptions, I am calling:
[self performSelectorInBackground:#selector(populateDB) withObject:nil];
The code for populateDb is below:
-(void)populateDB{
NSLog(#"start");
NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
NSManagedObjectContext *context;
if (coordinator != nil) {
context = [[NSManagedObjectContext alloc] init];
[context setPersistentStoreCoordinator:coordinator];
}
NSString *filePath = [[NSBundle mainBundle] pathForResource:#"input" ofType:#"txt"];
if (filePath) {
NSString * myText = [[NSString alloc]
initWithContentsOfFile:filePath
encoding:NSUTF8StringEncoding
error:nil];
if (myText) {
__block int count = 0;
[myText enumerateLinesUsingBlock:^(NSString * line, BOOL * stop) {
line=[line stringByReplacingOccurrencesOfString:#"\t" withString:#" "];
NSArray *lineComponents=[line componentsSeparatedByString:#" "];
if(lineComponents){
if([lineComponents count]==3){
float f=[[lineComponents objectAtIndex:0] floatValue];
NSNumber *number=[NSNumber numberWithFloat:f];
NSString *string1=[lineComponents objectAtIndex:1];
NSString *string2=[lineComponents objectAtIndex:2];
NSManagedObject *object=[NSEntityDescription insertNewObjectForEntityForName:#"Bigram" inManagedObjectContext:context];
[object setValue:number forKey:#"number"];
[object setValue:string1 forKey:#"string1"];
[object setValue:string2 forKey:#"string2"];
NSError *error;
count++;
if(count>=1000){
if (![context save:&error]) {
NSLog(#"Whoops, couldn't save: %#", [error localizedDescription]);
}
count=0;
}
}
}
}];
NSLog(#"done importing");
NSError *error;
if (![context save:&error]) {
NSLog(#"Whoops, couldn't save: %#", [error localizedDescription]);
}
}
}
NSLog(#"end");
}
Everything else is default core data code, nothing added.
I run that in the simulator.
I go to ~/Library/Application Support/iPhone Simulator/5.1/Applications//Documents
There is the sqlite file that is generated
I take that and I copy it in my bundle
I comment out the call to populateDb
I edit persistentStoreCoordinator to copy the sqlite file from bundle to documents at first run
- (NSPersistentStoreCoordinator *)persistentStoreCoordinator
{
#synchronized (self)
{
if (__persistentStoreCoordinator != nil)
return __persistentStoreCoordinator;
NSString *defaultStorePath = [[NSBundle mainBundle] pathForResource:#"myProject" ofType:#"sqlite"];
NSString *storePath = [[[self applicationDocumentsDirectory] path] stringByAppendingPathComponent: #"myProject.sqlite"];
NSError *error;
if (![[NSFileManager defaultManager] fileExistsAtPath:storePath])
{
if ([[NSFileManager defaultManager] copyItemAtPath:defaultStorePath toPath:storePath error:&error])
NSLog(#"Copied starting data to %#", storePath);
else
NSLog(#"Error copying default DB to %# (%#)", storePath, error);
}
NSURL *storeURL = [NSURL fileURLWithPath:storePath];
__persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:options error:&error])
{
NSLog(#"Unresolved error %#, %#", error, [error userInfo]);
abort();
}
return __persistentStoreCoordinator;
}
}
I remove the app from the simulator, I check that ~/Library/Application Support/iPhone Simulator/5.1/Applications/ is now removedI rebuild and launch again
As expected, the sqlite file is copied over to ~/Library/Application Support/iPhone Simulator/5.1/Applications//Documents
However the size of the file is smaller than in the bundle, significantly!
Also, doing a simple query with a predicate like this predicate = [NSPredicate predicateWithFormat:#"string1 == %#", string1]; clearly shows that string1 is not indexed anymore
Following that, I create a new version of the datamodel, with a meaningless update, just to force a lightweight migration
If run on the simulator, the migration takes a few seconds, the database doubles in size and the same query now takes less than a second to return instead of minutes.
This would solve my problem, force a migration, but that same migration takes 3 minutes on the iPad and happens in the foreground.
So hat's where I am at right now, the best solution for me would still be to prevent the indexes to be removed, any other importing solution at launch time just takes too much time.
Let me know if you need more clarifications...
Update 2
So the best result I have had so far is to seed the core data database with the sqlite file produced from a quick tool with similar data model, but without the indexes set when producing the sqlite file. Then, I import this sqlite file in the core data app with the indexes set, and allowing for a lightweight migration. For 2 millions record on the new iPad, this migration stills take 3 minutes. The final app should have 5 times this number of records, so we're still looking at a long long processing time.
If I go that route, the new question would be: can a lightweight migration be performed in the background?
Update
My question is NOT how to create a tool to populate a Core Data database, and then import the sqlite file into my app. I know how to do this, I have done it countless times. But until now, I had not realized that such method could have some side effect: in my case, an indexed attribute in the resulting database clearly got 'unindexed' when importing the sqlite file that way.
If you were able to verify that any indexed data is still indexed after such transfer, I am interested to know how you proceed, or otherwise what would be the best strategy to seed such database efficiently.
Original
I have a large CSV file (millions of lines) with 4 columns, strings and floats.
This is for an iOS app.
I need this to be loaded into core data the first time the app is loaded.
The app is pretty much non functional until the data is available, so loading time matters, as a first time user obviously does not want the app to take 20 minutes to load before being able to run it.
Right now, my current code takes 20 min on the new iPad to process a 2 millions line csv file.
I am using a background context to not lock the UI, and save the context every 1,000 records
The first idea I had was to generate the database on the simulator, then to copy/paste it in the document folder at first launch, as this is the common non official way of seeding a large database. Unfortunately, the indexes don't seem to survive such a transfer, and although the database was available after just a few seconds, performance is terrible because my indexes were lost. I posted a question about the indexes already, but there doesn't seem to be a good answer to that.
So what I am looking for, either:
a way to improve performance on loading millions of records in core data
if the database is pre-loaded and moved at first startup, a way to keep my indexes
best practices for handling this kind of scenario. I don't remember using any app that requires me to wait for x minutes before first use (but maybe The Daily, and that was a terrible experience).
Any creative way to make the user wait without him realizing it: background import while going through tutorial, etc...
Not Using Core Data?
...
Pre-generate your database using an offline application (say, a command-line utility) written in Cocoa, that runs on OS X, and uses the same Core Data framework that iOS uses. You don't need to worry about "indexes surviving" or anything -- the output is a Core Data-generated .sqlite database file, directly and immediately usable by an iOS app.
As long as you can do the DB generation off-line, it's the best solution by far. I have successfully used this technique to pre-generated databases for iOS deployment myself. Check my previous questions/answers for a bit more detail.
I'm just starting out with SQLite and I need to integrate a DB into one of my apps that will have a lot of indexed data in a SQLite database. I was hoping I could do some method where I could bulk insert my information into a SQLite file and add that file to my project. After discovering and reading through your question, the provided answer and the numerous comments, I decided to check out the SQLite source to see if I could make heads or tails of this issue.
My initial thought was that the iOS implementation of SQLite is, in fact, throwing out your indices. The reason is because you initially create your DB index on x86/x64 system. The iOS is an ARM processor, and numbers are handled differently. If you want your indexes to be fast, you should generate them in such a way that they are optimized for the processor in which they will be searched.
Since SQLite is for multiple platforms, it would make since to drop any indices that have been created in another architecture and rebuild them. However, since no one wants to wait for an index to rebuild the first time it is accessed, the SQLite devs most likely decided to just drop the index.
After digging into the SQLite code, I've come to the conclusion that this is most likely happening. If not for the processor architecture reason, I did find code (see analyze.c and other meta-information in sqliteint.h) where indices were being deleted if they were generated under an unexpected context. My hunch is that the context that drives this process is how the underlying b-tree data structure was constructed for the existing key. If the current instance of SQLite can't consume the key, it deletes it.
It is worth mentioning that the iOS Simulator is just that-- a simulator. It is not an emulator of the, hardware. As such, your app is running in a pseudo-iOS device, running on an x86/x64 processor.
When your app and SQLite DB are loaded to your iOS device, an ARM-compiled variant is loaded, which also links to the ARM compiled libraries within iOS. I couldn't find ARM specific code associated with SQLite, so I imagine Apple had to modify it to their suit. The could also be part of the problem. This may not be an issue with the root-SQLite code, it could be an issue with the Apple/ARM compiled variant.
The only reasonable solution that I can come up with is that you can create a generator application that you run on your iOS machine. Run the application, build the keys, and then rip the SQLite file from the device. I'd imagine such a file would work across all devices, since all ARM processors used by iOS are 32-bit.
Again, this answer is a bit of an educated guess. I'm going to re-tag your question as SQLite. Hopefully a guru may find this and be able to weigh in on this issue. I'd really like to know the truth for my own benefit.

Resources