At first I have a plist file in app bundle. At some point the file can get updated (downloaded into Documents Folder). Whats the best way to know which file is newer? The mainbundle plist could get updated with new App Version and then the downloaded one would be the oldest.
I do have a timestamp inside the plist but I actually don't want to load each of them into memory to be able to compare the dates, as each takes 1-2 seconds.
Is there some kind of creation date that I could compare?
How about this:
NSError *error = nil;
NSDictionary* dict = [NSFileManager attributesOfItemAtPath:path error:&error];
NSDate* date = [dict fileModificationDate];
You can use -[NSFileManager attributesOfItemAtPath:error:] to get the filesystem-level attributes of each plist, then compare values for modification dates (using the NSFileModificationDate string constant).
Related
In my existing app, I have about 450 very small PDF files that are lyrics to songs. They are all stored in a directory called (appropriately) "ThePDFs". The downside with this is that every time I want to add a new set of lyrics, I have to release an update to the app. What I'd love to do is use something like the Parse server to point toward PDFs stored online, and allow the user to select those and add them to their existing list.
However, I don't think it is possible or allowed for me to take an outside PDF and write them into that directory called "ThePDFs"? If it's not allowed, what would be a good way to accomplish this, in such a way that all the songs are in one list? I know I could download them into a documents folder for the app, but then I have songs in multiple locations, and accessing them all in one tableview might be more difficult. I suppose at first launch of the app, I could have all the PDFs copy from the bundled section into the documents directory, and search to it from that?
I know I have a lot of options, I'm just curious if there's something I'm missing.
UPDATE
So, I'm trying to do the first option of leaving PDFs where they are, and make a PLIST that would have an item with each song's name, along with the location for where it is in the file system. Then, when new songs are added, it would do the same, and I could read from the PLIST file to get all songs and be able to open them, no matter where they are. My issue right now is getting the initial PLIST set up, and the logic behind what I would do for each subsequent addition. The issue I get is
NSDictionary initWithObjects:forKeys:]: count of objects (0) differs from count of keys (1)'
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsDirectory = [paths objectAtIndex:0];
NSString *path = [documentsDirectory stringByAppendingPathComponent:#"mastersonglist.plist"];
NSFileManager *fileManager = [NSFileManager defaultManager];
if (![fileManager fileExistsAtPath: path]) {
path = [documentsDirectory stringByAppendingPathComponent: [NSString stringWithFormat:#"mastersonglist.plist"] ];
}
NSError *error = nil;
NSMutableDictionary *data;
if ([fileManager fileExistsAtPath: path]) {
data = [[NSMutableDictionary alloc] initWithContentsOfFile: path];
}
else {
// If the file doesn’t exist, create an empty dictionary
data = [[NSMutableDictionary alloc] init];
}
NSBundle *bundle = [NSBundle mainBundle];
self.files = [bundle pathsForResourcesOfType:#"pdf" inDirectory:#"thepdfpowerpoints"];
NSMutableArray *names = [NSMutableArray arrayWithCapacity:[self.files count]];
for (NSString *path in self.files) {
[names addObject:[[path lastPathComponent] stringByDeletingPathExtension]];
}
for (NSString *test in names) {
// [data setObject:names forKey:#"thename"];
// [data writeToFile:path atomically:YES];
NSDictionary *innerDict;
NSLog(#"NAMES%#", names);
innerDict = [NSDictionary dictionaryWithObjects:
[NSArray arrayWithObjects: names, path, nil]
forKeys:[NSArray arrayWithObjects:#"Names", #"Path", nil]];
NSLog(#"DICT%#", innerDict);
NSMutableDictionary *plistdictionary = [[NSMutableDictionary alloc]initWithContentsOfFile:path];
NSMutableArray *notes=[plistdictionary objectForKey:#"Questions"];
//NSLog(#"QUESTION %#", innerDict);
[notes addObject:innerDict];
NSLog(#"NOTES %#", notes);
NSDictionary *outerDict = [NSDictionary dictionaryWithObjects:
[NSArray arrayWithObjects: notes, nil]
forKeys:[NSArray arrayWithObjects:#"Questions", nil]];
id plist = [NSPropertyListSerialization dataFromPropertyList:(id)outerDict
format:NSPropertyListXMLFormat_v1_0 errorDescription:&error];
[plist writeToFile:path atomically:YES];
}
First, you're correct, you cannot add files to the app bundle at run-time.
So, various ways to approach this, depending on what functionality you want to provide to the user.
First, you could:
include the existing PDFs in your app bundle
check a remote server for new PDFs on launch
download the new PDFs into a local app folder
generating a data list from both locations to display in a single table would really be a pretty straight-forward and not-at-all complex task.
Second, you could:
include NO PDFs in your app bundle
check a remote server for new PDFs on launch (on first launch they'd all be new)
download the PDFs (on a background thread, of course) into a local app folder
Now you only need to manage the files in one location.
Third, you could:
include NO PDFs in your app bundle
check a remote server for the list of available PDFs on launch
retrieve the PDFs on-demand when the user selects them
optionally "cache" them locally - so only retrieve if the PDF has not already been selected / downloaded
Edit
You don't really need to know the file locations...
I don't know what information you have associated with each PDF file, so, to simplify, let's assume you're displaying a list of PDF file names (no other data).
The general idea could be:
get the list of PDF files in the bundle
append the list of PDF files in the downloads folder
sort the full list alphabetically
user selects a PDF
does file downloadsPath/filename.pdf exist?
Yes? load it
No? load it from bundle
Same principal if you have some sort of data file - .plist, csv, json, whatever - that would have maybe filename, song title, artist, date, etc.
load the data for the PDF files included in the bundle
append the data for the PDF files in the downloads folder
sort the full list alphabetically
user selects a song
does file downloadsPath/filename.pdf exist?
Yes? load it
No? load it from bundle
No need to track the file location in your data -- just find it when you want to load it.
I've got a Core Data model with 15 versions. It's got code to progressively migrate from the current store's version to the latest version on launch.
Key to that is a call to
NSDictionary* options = #{ NSMigratePersistentStoresAutomaticallyOption : #true,
NSInferMappingModelAutomaticallyOption : #true };
NSDictionary* sourceMetadata = [NSPersistentStoreCoordinator
metadataForPersistentStoreOfType: inType
URL: inSourceStore
options: options
error: outError];
NSManagedObjectModel* model = [NSManagedObjectModel mergedModelFromBundles: #[ [NSBundle bundleForClass: [self class]] ]
forStoreMetadata: inSourceMetadata];
But that's always returning nil, and I'm not sure why. The existing store is version 14, the new model is version 15.
Now, the last change to the model was fairly trivial (the addition of a couple optional fields), so I would have thought it could infer the mapping automatically, but that wasn't working, so I added a mapping model from version 14 to version 15 using Xcode's assistant for that, and made no changes.
Any idea why it's returning nil, or what I can do to investigate this further?
In the same vein, when I say "version 14", I'm referring to the sequential numbering of the .xcdatamodel files. Is there any way to look at the actual store and determine which version of the model Core Data thinks it is?
Well, first of all, you seem like you know what you are doing, having survived through 14 Core Data migrations and all. So I think you should be on the lookout for some silly forehead-slapping type of mistake.
Ensure that [NSBundle bundleForClass: [self class]] is returning the expected bundle, which contains a directory Contents/Resources/YourModelName.momd, and that this directory contains all of the required .mom files (one for each version), and a VersionInfo.plist file. My builds also contain a .omo file for the latest version only.
Now I shall answer your second question, which indeed may help you to answer your first question.
In that VersionInfo.plist file you will find a dictionary named NSManagedObjectModel_VersionHashes, which in turn contains sub-dictionaries, one key for each version. Each version sub-dictionary contains a key for each of your entity names and value which is a 32-byte (256 bit) hash of the attributes and relationships of that entity in that version. Let's call this these the model hashes.
Now open up a store database file with a SQLite viewer, or the sqlite3 command line tool. In that database, alongside one table for each of the entities in the model, you will see a table named Z_METADATA with one row and three columns. The value of the column named Z_PLIST is typed as a blob of binary data. Copy that data to a file, name it with extension .plist, doubleclick and, surprise it opens in your favorite plist editor because that data is in fact a string of text representing an Apple property list in XML format. The value of its key NSStoreModelVersionHashes is in fact a sub-dictionary which is just like the sub-dictionary in the VersionInfo.plist file. Let's call this the store hashes. The 32-byte (256-bit) version hash is Base64 encoded. (There are 44 Base64 characters. Since each Base64 character represents 6 bits, 44 characters can represent up to 44*6 = 264 bits.)
Finally, to answer your second question, the storeMetadata passed to +[NSManagedObjectModel mergedModelFromBundles:forStoreMetadata:] is in fact the Z_METADATA from the store, which contains those store hashes. +[NSManagedObjectModel mergedModelFromBundles:forStoreMetadata:] compares these store hashes to the model hashes from each candidate data model in the passed-in bundle, and returns the model whose model hashes match the store hashes for all entities, with no extra unmatched entities on either side.
So you see it's kind of tedious to do the comparison manually. But probably while spelunking through these plists you will find that forehead-slapper. If not, give us some more context around that code you pasted and probably someone can help.
Ah. It turns out, I edited the latest model version, rather than adding a new one. That's why none would match. Once I reverted the latest version and added a new model version with the changes, it worked fine, even without the default mapping models.
I figured this out by testing each model to see if it matched against the source metadata, using the following:
NSDictionary* storeHashes = [sourceMetadata objectForKey: NSStoreModelVersionHashesKey];
NSArray<NSURL*>* urls = [self getModelURLs];
urls = [urls sortedArrayUsingComparator:
^NSComparisonResult(id _Nonnull obj1, id _Nonnull obj2)
{
NSURL* s2 = obj1;
NSURL* s1 = obj2;
return [s1.lastPathComponent compare: s2.lastPathComponent options: NSNumericSearch];
}];
for (NSURL* url in urls)
{
NSDictionary* modelHashes = [self getHashesForModelAtURL: url];
// Compare the hashes…
bool matches = true;
for (NSString* entityKey in storeHashes.allKeys)
{
NSString* storeHash = storeHashes[entityKey];
NSString* modelHash = modelHashes[entityKey];
if (![storeHash isEqual: modelHash])
{
NSLogDebug(#"Model %# has mismatch on %#", url.lastPathComponent, entityKey);
matches = false;
}
}
if (matches)
{
NSLogDebug(#"Version matches: %#", url.lastPathComponent);
break;
}
}
- (NSArray<NSURL*>*)
getModelURLs
{
NSBundle* bundle = [NSBundle bundleForClass: [self class]];
NSArray<NSURL*>* urls = [bundle URLsForResourcesWithExtension: #"mom" subdirectory: #"Model.momd"];
return urls;
}
- (NSDictionary*)
getHashesForModelAtURL: (NSURL*) inURL
{
NSManagedObjectModel* model = [[NSManagedObjectModel alloc] initWithContentsOfURL: inURL];
NSDictionary* hashes = model.entityVersionHashesByName;
return hashes;
}
I am building a mobile iOS app for a web backend. I retrieve the JSON response using the following code:
NSError *error;
NSString *url_string = [NSString stringWithFormat: #"https://myURL"];
NSData *data = [NSData dataWithContentsOfURL: [NSURL URLWithString:url_string]];
NSDictionary *jsonDict = [NSJSONSerialization JSONObjectWithData:data options:NSJSONReadingMutableContainers error:&error];
What would be the best/simplest way to store this data locally so that I can use a local database when internet connectivity is not available? The intention is to check web connectivity at launch of the app. If connection is available, get JSON response, parse and update local copy, if connection is not available parse the data from local storage.
Thanks in advance.
If you wish to cache the JSON data so you can still use it offline, I would write the JSON dictionary (or the data) to a file in your app's sandbox. A subfolder of the "Application Support" folder would be a good place. You don't want to use the Caches folder in this case because the files could be purged by iOS when you need them offline.
The trick is to map a given URL to a filename. You need this mapping to both save a file for a given URL and to later load the file if offline. You should be able convert a URL to a useful filename simply by converting all / characters to something else such as an underscore.
You probably don't want these files backed up when a user backups their iOS device so be sure you mark the files with the "do not backup" attribute. There are many existing question covering that topic.
The best way is CoreData and
the simplest way is NSUserDefaults
NSUserDefaults Class Reference
[[NSUserDefaults standardUserDefaults] setObject: jsonDict forKey:#"dictionaryKey"];
//...
NSDictionary * myDictionary = [[NSUserDefaults standardUserDefaults] dictionaryForKey:#"dictionaryKey"];
How can I find out when a file was last time modified in Cocoa?
I tried using NSFileModificationDate attribute of an NSFile but the modification date gets update when you read a file. I just want to know when it was last time changed like in the Mac OS X Finder.
You can try next code:
NSDate *fileModificationDate = nil;
[fileUrl getResourceValue:&fileModificationDate forKey:NSURLContentModificationDateKey error:nil];
NSLog(#"modification date %#", fileModificationDate)
You can see the resource keys in NSURL.h - NSURLContentModificationDateKey and other.
Hope it will help.
Conclusion
Problem closed, I think.
Looks like the problem had nothing to do with the methodology, but that the XCode did not clean the project correctly in between builds.
It looks like after all those tests, the sqlite file that was being used was still the very first one that wasn't indexed......
Beware of XCode 4.3.2, I have nothing but problems with Clean not cleaning, or adding files to project not automatically being added to the bundle resources...
Thanks for the different answers..
Update 3
Since I invite anybody to just try the same steps to see if they get the same results, let me detail what I did:
I start with blank project
I defined a datamodel with one Entity, 3 attributes (2 strings, 1 float)
The first string is indexed
In did finishLaunchingWithOptions, I am calling:
[self performSelectorInBackground:#selector(populateDB) withObject:nil];
The code for populateDb is below:
-(void)populateDB{
NSLog(#"start");
NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
NSManagedObjectContext *context;
if (coordinator != nil) {
context = [[NSManagedObjectContext alloc] init];
[context setPersistentStoreCoordinator:coordinator];
}
NSString *filePath = [[NSBundle mainBundle] pathForResource:#"input" ofType:#"txt"];
if (filePath) {
NSString * myText = [[NSString alloc]
initWithContentsOfFile:filePath
encoding:NSUTF8StringEncoding
error:nil];
if (myText) {
__block int count = 0;
[myText enumerateLinesUsingBlock:^(NSString * line, BOOL * stop) {
line=[line stringByReplacingOccurrencesOfString:#"\t" withString:#" "];
NSArray *lineComponents=[line componentsSeparatedByString:#" "];
if(lineComponents){
if([lineComponents count]==3){
float f=[[lineComponents objectAtIndex:0] floatValue];
NSNumber *number=[NSNumber numberWithFloat:f];
NSString *string1=[lineComponents objectAtIndex:1];
NSString *string2=[lineComponents objectAtIndex:2];
NSManagedObject *object=[NSEntityDescription insertNewObjectForEntityForName:#"Bigram" inManagedObjectContext:context];
[object setValue:number forKey:#"number"];
[object setValue:string1 forKey:#"string1"];
[object setValue:string2 forKey:#"string2"];
NSError *error;
count++;
if(count>=1000){
if (![context save:&error]) {
NSLog(#"Whoops, couldn't save: %#", [error localizedDescription]);
}
count=0;
}
}
}
}];
NSLog(#"done importing");
NSError *error;
if (![context save:&error]) {
NSLog(#"Whoops, couldn't save: %#", [error localizedDescription]);
}
}
}
NSLog(#"end");
}
Everything else is default core data code, nothing added.
I run that in the simulator.
I go to ~/Library/Application Support/iPhone Simulator/5.1/Applications//Documents
There is the sqlite file that is generated
I take that and I copy it in my bundle
I comment out the call to populateDb
I edit persistentStoreCoordinator to copy the sqlite file from bundle to documents at first run
- (NSPersistentStoreCoordinator *)persistentStoreCoordinator
{
#synchronized (self)
{
if (__persistentStoreCoordinator != nil)
return __persistentStoreCoordinator;
NSString *defaultStorePath = [[NSBundle mainBundle] pathForResource:#"myProject" ofType:#"sqlite"];
NSString *storePath = [[[self applicationDocumentsDirectory] path] stringByAppendingPathComponent: #"myProject.sqlite"];
NSError *error;
if (![[NSFileManager defaultManager] fileExistsAtPath:storePath])
{
if ([[NSFileManager defaultManager] copyItemAtPath:defaultStorePath toPath:storePath error:&error])
NSLog(#"Copied starting data to %#", storePath);
else
NSLog(#"Error copying default DB to %# (%#)", storePath, error);
}
NSURL *storeURL = [NSURL fileURLWithPath:storePath];
__persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:options error:&error])
{
NSLog(#"Unresolved error %#, %#", error, [error userInfo]);
abort();
}
return __persistentStoreCoordinator;
}
}
I remove the app from the simulator, I check that ~/Library/Application Support/iPhone Simulator/5.1/Applications/ is now removedI rebuild and launch again
As expected, the sqlite file is copied over to ~/Library/Application Support/iPhone Simulator/5.1/Applications//Documents
However the size of the file is smaller than in the bundle, significantly!
Also, doing a simple query with a predicate like this predicate = [NSPredicate predicateWithFormat:#"string1 == %#", string1]; clearly shows that string1 is not indexed anymore
Following that, I create a new version of the datamodel, with a meaningless update, just to force a lightweight migration
If run on the simulator, the migration takes a few seconds, the database doubles in size and the same query now takes less than a second to return instead of minutes.
This would solve my problem, force a migration, but that same migration takes 3 minutes on the iPad and happens in the foreground.
So hat's where I am at right now, the best solution for me would still be to prevent the indexes to be removed, any other importing solution at launch time just takes too much time.
Let me know if you need more clarifications...
Update 2
So the best result I have had so far is to seed the core data database with the sqlite file produced from a quick tool with similar data model, but without the indexes set when producing the sqlite file. Then, I import this sqlite file in the core data app with the indexes set, and allowing for a lightweight migration. For 2 millions record on the new iPad, this migration stills take 3 minutes. The final app should have 5 times this number of records, so we're still looking at a long long processing time.
If I go that route, the new question would be: can a lightweight migration be performed in the background?
Update
My question is NOT how to create a tool to populate a Core Data database, and then import the sqlite file into my app. I know how to do this, I have done it countless times. But until now, I had not realized that such method could have some side effect: in my case, an indexed attribute in the resulting database clearly got 'unindexed' when importing the sqlite file that way.
If you were able to verify that any indexed data is still indexed after such transfer, I am interested to know how you proceed, or otherwise what would be the best strategy to seed such database efficiently.
Original
I have a large CSV file (millions of lines) with 4 columns, strings and floats.
This is for an iOS app.
I need this to be loaded into core data the first time the app is loaded.
The app is pretty much non functional until the data is available, so loading time matters, as a first time user obviously does not want the app to take 20 minutes to load before being able to run it.
Right now, my current code takes 20 min on the new iPad to process a 2 millions line csv file.
I am using a background context to not lock the UI, and save the context every 1,000 records
The first idea I had was to generate the database on the simulator, then to copy/paste it in the document folder at first launch, as this is the common non official way of seeding a large database. Unfortunately, the indexes don't seem to survive such a transfer, and although the database was available after just a few seconds, performance is terrible because my indexes were lost. I posted a question about the indexes already, but there doesn't seem to be a good answer to that.
So what I am looking for, either:
a way to improve performance on loading millions of records in core data
if the database is pre-loaded and moved at first startup, a way to keep my indexes
best practices for handling this kind of scenario. I don't remember using any app that requires me to wait for x minutes before first use (but maybe The Daily, and that was a terrible experience).
Any creative way to make the user wait without him realizing it: background import while going through tutorial, etc...
Not Using Core Data?
...
Pre-generate your database using an offline application (say, a command-line utility) written in Cocoa, that runs on OS X, and uses the same Core Data framework that iOS uses. You don't need to worry about "indexes surviving" or anything -- the output is a Core Data-generated .sqlite database file, directly and immediately usable by an iOS app.
As long as you can do the DB generation off-line, it's the best solution by far. I have successfully used this technique to pre-generated databases for iOS deployment myself. Check my previous questions/answers for a bit more detail.
I'm just starting out with SQLite and I need to integrate a DB into one of my apps that will have a lot of indexed data in a SQLite database. I was hoping I could do some method where I could bulk insert my information into a SQLite file and add that file to my project. After discovering and reading through your question, the provided answer and the numerous comments, I decided to check out the SQLite source to see if I could make heads or tails of this issue.
My initial thought was that the iOS implementation of SQLite is, in fact, throwing out your indices. The reason is because you initially create your DB index on x86/x64 system. The iOS is an ARM processor, and numbers are handled differently. If you want your indexes to be fast, you should generate them in such a way that they are optimized for the processor in which they will be searched.
Since SQLite is for multiple platforms, it would make since to drop any indices that have been created in another architecture and rebuild them. However, since no one wants to wait for an index to rebuild the first time it is accessed, the SQLite devs most likely decided to just drop the index.
After digging into the SQLite code, I've come to the conclusion that this is most likely happening. If not for the processor architecture reason, I did find code (see analyze.c and other meta-information in sqliteint.h) where indices were being deleted if they were generated under an unexpected context. My hunch is that the context that drives this process is how the underlying b-tree data structure was constructed for the existing key. If the current instance of SQLite can't consume the key, it deletes it.
It is worth mentioning that the iOS Simulator is just that-- a simulator. It is not an emulator of the, hardware. As such, your app is running in a pseudo-iOS device, running on an x86/x64 processor.
When your app and SQLite DB are loaded to your iOS device, an ARM-compiled variant is loaded, which also links to the ARM compiled libraries within iOS. I couldn't find ARM specific code associated with SQLite, so I imagine Apple had to modify it to their suit. The could also be part of the problem. This may not be an issue with the root-SQLite code, it could be an issue with the Apple/ARM compiled variant.
The only reasonable solution that I can come up with is that you can create a generator application that you run on your iOS machine. Run the application, build the keys, and then rip the SQLite file from the device. I'd imagine such a file would work across all devices, since all ARM processors used by iOS are 32-bit.
Again, this answer is a bit of an educated guess. I'm going to re-tag your question as SQLite. Hopefully a guru may find this and be able to weigh in on this issue. I'd really like to know the truth for my own benefit.