I have a data set of 1.2 million key-value pairs. The keys are a string (a sequence of numbers up to 22 characters in length), and the values are strings.
What's the best way to ship this so a value can be looked up and retrieved quickly?
I suspect a plist is not the way to go for a data set this size.
I have the data set stored in two ways - a CSV, and a mySQL database table with 2 columns. I
ll go forward using whatever method gets the data into the app the best.
Core Data and SQLite are two good options for dealing with very large data sets in iOS. It's not difficult to create a Core Data model for the kind of data you're talking about. You can then copy that model into a little command line program that you'll write to move the data into a Core Data store. You can then move the resulting data file into your iOS app's resources.
A third option, particularly useful if the data is likely to change often, is to build a web service to provide the data from the service. I don't think this is what you're asking about, but it's something to consider if the data set is very large and/or subject to frequent change.
a collection of text files could work well. you can:
divide them into multiple files (e.g. by leading character range).
order pairs appropriately (e.g. by character number)
and quickly/easily read incrementally/portions as appropriate.
balance resources between file reads and memory usage well.
choosing the right encoding for the strings can also help (i'd start with utf8 if it's mostly in ascii).
if distribution size is also your concern, you could compress/decompress these files.
or you can just take this approach and use a custom serialized class to represent a subsets of the collection if that sounds like too much parse and read implementation.
if you're using objc types for storage and/or parsing, then it would be good to keep those files small. if you use c or c++, then it would help to profile the app.
your data set will require up to 30 MB using an 8 bit per character single byte encoding. one large file (again, ordered) which you mmap would also be worth consideration. see [NSData initWithContentsOfMappedFile:path];.
My personal experience is with a plist file that has only thousands of records and I can say that it isn't it so fast. So my options for this amount of data you have are:
A database.
Or if you have a sorting criteria for those keys and prefer plist
files split it in many files and keep a reference dictionary with the start key of every file. For ex. all keys that begin with 'abc' go in a.plist etc.
(I don't know if it's the case with your app but you can consider moving the data to a server and search via a webservice, specifically if your data will grow.)
A sqlite file is probably your best bet. You can create it on the desktop using either the command-line sqlite3 or any sqlite gui. Make sure you index the key column.
Import the csv file as described here: Importing csv files into sqlite
Then just add the database to your project/target. If you want to modify the database at runtime, however, you'll have to copy it out into your Documents or cache directories.
For an objective-c wrapper around sqlite, I like fmdb
Related
I am working on my first iOS app which will be deployed for both iPhones and iPads. The app contains data that needs to be bundled with the app, which will be used when the device will be offline.
The offline version has atleast 35-40 records with each record containing images(which would be bundled in the app, only names will be saved), and a varchar field which would be atleast 1000 words and a boolean field.
I have found three possible solution for the same
Save all the fields using database (SQlite or Coredata), however
I am concerned about the table which will have 1000 words. But since
the varchar field might vary, I need to allocate max 2000 (or more,depending on the actual length of keywords)
limit (which will lead unnecessary allocation of memory resources)
Another Approach I would like to have is save the information in form of json locally and use it as and when required and save the boolean fields(only true locally in NSUserDefaults)
Use the JSON Approach as discussed above and create a database for managing the boolean fields.
I would like to seek the opinion of StackOverflow community on what would be ideal/optimized approach for this scenario. Also, I am open for any other approach as well.
Edit 1
Proposed provisional databased structure
Listing Table
id -> int (autoincrement)
name -> varchar(25)
imagename -> varchar(10)
desription -> varchar(2000)
favorite -> boolean
It sounds as though the text field (with the 1000-2000 words) is static text that is bundled with the app and can not be changed by the user of the app. If that's the case, then you can store that data in the app bundle with plist files, or JSON files and load it on demand (assuming you don't need to search though it).
Then, if each of those records has only a single boolean value that is changeable by the user, those could be stored in NSUserDefaults very easily (since you've stated you're only dealing with 35-40 records). You'd use the id to link the boolean to the data file.
You could use Core Data or Realm to store the data, but it may be overkill if you don't need a search feature and the user can't change the text. But if you do go with a database option, be aware that you can not store static data (the text), in a location that is backed up by iCloud, or Apple will reject your app. Regardless of whether you use iCloud in the app or not. So if you were to create a Core Data persistent store and save it to the users Documents folder, then load in all the static data, you will be rejected. You would want to save that data store in the users Cache folder so that iCloud doesn't back it up. The issue you'll hit after that though is that you want the user's choices that are your boolean values backed up. This means they need to be stored in a different place. Core Data does have a feature that lets you create Configurations where it will separate the user changeable data from the non-changeable data, but again, that's overkill for your case.
I'd recommend starting with Realm over 'Core Data` for such a small dataset. It's much easier to get up and running.
If you need to look into the fields, CoreData is th best approach, because you can easily access your data using NSPredicates, ( Like an SQL where statement` ).
But if you need to load everything at each launch, you can just store everything to a file ( plist, son ... ), because it is really more easy to managed, and to update ( If you update CoreData Model, the change may be complicated on the App update ).
So my short answer is :
If you need to teach into your data => Core Data
Else => File on local Storage
Do not use UserDefault to achieve that, this is not designed for it.
It is depends on you data base's complexity and operation.
See, first thing is whatever data base system you used, there is no performance difference.
Every database system has different complexity and operation limitations.
For example, It is very simpler to use NSUserDefaults to store and retrieve data.
But if you required to do relational operation between bulk data then it is better to use sqlite or core data. Relational database operations are easily performed by sqlite or coredata compare to others.
There is another option is property - list also available if you data is only type of key value pair.
Core data is totally based on sqlite. In root core data itself using sqlite.
Difference between core data and sqlite is : core data provides more flexibility to use it but it is comparatively hard or complex to learn. Where sqlite not provide flexibility compare to core data but it is less complex to learn and use. Flexibility means for example : you can see visual representation of core data. Can visually add entity or attributes etc.
So, select data base as your need and complexity of use or base on operations that you will required to perform. Your database is not much big and not complex and not required any relational operations or multiple tables then you can use user defaults or property list also.
Hope this will help :)
Pitching in with an option.
To make it easy for yourself later on, I would suggest using CoreData. This lets you easily manage the products for reading and writing. This also gives you a good persistent storage.
To the description issue; you could store the description for each product in its own file with a unique name that references the product you store in CoreData. In your CoreData entity you define a descriptionFile, which will hold the file path.
This approach makes it easier for CoreData when you fetch the objects, maybe you want a browse view where you don't need to display the description, therefore you don't need to load the description text into memory. When a product is selected, load the description file for that product and display the text within.
Happy Coding :)
I have created an app using ionic and cordova and now I want to remake it on iOS. I am working with iOS for the first time, and I cannot figure out how to store data.
For example: I have a form where user has to input some data, but the inputs are not in one view, there must be several views. I used to create empty array and just put everything step by step, but now i can't use same view controller on multiple views. Tried to do it with core data, but core data cannot store arrays. My object would look something like this:
var sampleArray = (
duration: 13,
dayOfTheWeek: Thursday,
personList: [
(name: Rocky,
age: 26),
(name: Ralph,
age:23)
]
)
The question would be: How could I make an input form which would be on several views and where should I store the data, and later I would be able to store all the data into core data?
You can work with persistent data in several ways on iOS.
User Default
This is a tool that is used to store small amounts of information like user settings, preferences etc. Don't use it for data that will scale with application usage (e.g. like notes in notepad app). Documentation will answer all your questions about User Defaults.
Database
You have Core Data as an out of the box solution which is build on top of the SQLite and takes some time to learn, but from my experience it's worth the effort. You are free to use pure SQLite or other database type, but it requires more code and probably custom frameworks.
Text files
You can use arbitrary XML, JSON or CSV files to store your data. Tooling is rich (e.g. NSXMLParser or SwifyJSON just to name two) and if you look on Github, you will find what you need. You can also use build in combination of NSCoder and NSKeyArchiver / NSKeyUnarchiver which are easy to grasp.
Binary files
Finally, for a local storage you can use binary files i.e. images. This is too advanced topic to cover here, but I want to share an example of Open Raster file format. It is used to save informations for drawing apps (eq. GIMP) and inside, it is basically an XML file and a bunch of images compressed to zip and named as .ora file. Creating your own specification for a hybrid format is not that hard.
Network repository
Just to not overlook other methods, you can use remote database API to store data outside of the device, but of course you need your own host and some backend skills.
I hope I didn't miss something important. I just wanted to sum up this knowledge in one place for future reference.
As the first comment says, your question is quite large.
When you say 'one form on several view', I consider it as 'one form per view'.
Keep It Simple S... ;)
(Except if you use page control for your form.)
Basically, you have three ways to store data :
NSUserDefaults :
Store data in Dictionary for later use
File :
Save data to a File (why not .csv like ?)
CoreData :
You can persist arrays as binary data in Core Data
There are numerous tutorials on these topics.
www.raywenderlich.com site is a good one to begin...
I've used archiving, user defaults, and some Core Data in my apps before, but I'm running into a wall re: the best method for my current scenario. I have an app that needs to instantiate some objects from a resource file each time before it's used. An example of one of the objects could be a "MathQuestion" object that has the properties:
questionID (Int) - 2341
questionText (String) — "What is the square root of _?"
questionVariable (Float) – "4"
correctAnswer (a block/closure that returns a Float) – "{return sqrt(value)}".
Ideally, I'd just have something like a spreadsheet with columns for each of these properties and rows for each of the different questions. That way, I would really be able to visualize all the data and make quick changes during development.
My app uses Parse, which is great for visualization and easy editing of values, but for this case, I'd rather the resources remained on the device and not the Parse server. I've been considering Core Data up till this point, but (this could be my inexperience with C.D.), I'm unaware of any way to manually edit the data—and it seems like it may be overkill for what I'm looking for anyways. (I basically just need a way to upload and parse a CSV!) Any advice would be welcome!
I would still recommend Core Data. It is simply the most efficient and scalable mechanism to store and retrieve data.
When I work in your kind of scenario, what I often do is work with a spreadsheet where I can conveniently edit the data. You could edit a CSV version of it that you have included in your target. (If this does not work for you, you could also copy paste into a separate CSV file right into Xcode.)
On every start, you just trash the data store (using NSFileManager when creating the NSPersistentStoreCoordinator). Then you call a method that reads in the CSV and stores it in Core Data.
Once you are done with development, you simply keep the sqlite file (you can include it in the bundle and copy it over, or re-generate it from your CSV the first time the app runs).
I have a simple problem to solve but want to know which is better pattern to use and understand the reason for the choice.
Problem :
I want to create a utility which developers can use to check whether the feature should be enabled/disabled depending on the server package version.
eg : server package versions like 10.234, 11.1 etc and client versions 9.3,10.2 etc
Validation on client to see min version on server package for feature to be enabled/disabled.
example data would be like "search lookup feature >= 10.234". And sometimes complex situation need to even check client version whether the feature should be supported.
edit:
Note: Application is very huge and memory is full for most of the time. (thousands of records of organisation data.) So memory is bottle neck.
Just it even flashed to mind to used macro as to do all comparisons and returning value.
I think Plist would be heavy as all the objects would be in NSDictionary and even to access one object, I'll be holding all data.
I want to reduce memory overhead and comparisons too.
For Light data, Use NSUserDefaults or PList. SQLite and Core Data is overkilled.
Updated Answer for Updated Question:-
For server package versions/your app version or other light weight data, you can always use NSUserDefaults or PList.
For records of organisation data, you might want to consider Core Data.
Yes, you can use 2 different types of persistent storage inside your app.
If your data is never going to change, or grow you better use NSUserDefaults, plist or csv file. If you think that user should have the ability to change or add new entries to this data file you better go ahead with CoreData
Plists, csv, and flat files have low overhead for small to medimum amounts of data (and for data where you use all or nearly all) of it.
Systems that only use some of the data on any given run AND have large amounts of data AND have the data structured so SELECT can use an index can be faster (lots and lots faster) with a database (SQLite, or CoreData with the SQL Persistant store).
As a gut feel I expect your problem could be handled well by SQL, but only if you are talking about 1000s of configurations not 10s or even 100s...but you would be better served by benchmarks then my guesses. Go code :-)
I have a CoreData database that gets initialized with a local file.
The CoreData schema that looks like this:
Category -->> Objections -->> Responses -->> Evidence
("-->>" means, has many)
Each entity also has a description that can be anywhere from 2 to thousands of characters long, stored in an NSString.
Question: How can I store this data so that it would be easy for someone to edit without having to know a lot about programming? (But also follow best practices)
Currently, I am thinking of these as possible approaches:
1) Store everything in 1 big plist file. This would be about 25 pages long.
2) Separate each entity into it's own PList file, and relate each the values with an ID#, like a Relational Data Base. This would make the files a more manageable size, but you have to keep track of ID#.
3) Same as above, but with JSON
Create a dead simple desktop application that uses Core Data. Let the people edit the file in that desktop application and keep it stored in Core Data. Then when you ship your application you can embed that SQLite file into your iOS application so there is no start up parsing required.
Standing up an OS X app for this that doesn't need to be pretty is dead simple. Takes an afternoon or two at most and saves you a ton of headaches.
run in simulator, do the parsing of the plist OR CSV (might be easier, since thats an excel compatible format) there (in the simulated app) and then copy the resulting DB to your xcode project and ship it.
=> your 'content provider' can work with excel
=> you don't have to ship a CSV (or whatever file you use) since you have a filled DB after you parsed it in the simulator
There is a nice post from Mattt Thompson about Core Data Libraries & Utilities: http://nshipster.com/core-data-libraries-and-utilities/, maybe something fit for you. =]