I am developing an iOS app with objective-C. With my 'GET' request am getting a large chunk of json data in below format.
[ { #"value": #1,#"day":#2,#"hour":#1} , {#"value": #1,#"day":#1,#"hour":#1 }....]
Note, This array always contain 168 fixed number objects.
Inside my app I have different UI controls which suppose to show different chunks of the obtained data. For example clicking 'Button1' suppose to show ob1---obj10 and so on.
In theory its all working, but I am not happy with my design approach.
Because for each button press I am calling the api to get the entire data set again and extracting the required data.
Ideally what I think should happen is I should store the data locally upon first 'GET' request and different classes within my app should be able to extract required information.
Same method should apply to my 'POST' requests. I am confused with what options I have, and what is best practice in this situation. I can think of following
Store data in an Array ?
Store the data in adatabase like sqlite ?
Finally plists ?
Using core data is bit of an overkill ?
168 records where each is roughly 40 bytes gives about 7 Kb of data, which is probably reduced down to something like 2 Kb if your API server supports gzip compression. This is nothing according to modern web standards.
You could download it completely, parse and save to a variable in your view controller. I recommend creating a "model" class (as in MVC) to wrap your data nicely and factor out the rows selection logic.
CoreData (or SQLite) is definitely an overkill for this amount of data.
You can use something like a plist or NSKeyedArchiver to cache the data to disk if you need offline support, i.e. you want to have an app working without an internet connection.
I'm using NSURLConnection to interact with the server side and I have observed that when the server take time to respond the system allows about 40 mo.
I don't know if I'm the only one to have this problem.
thanks in advance.
Yes this is possible in case if your data for response is large in size. Generally what we do, we create instance of NSData and append all downloaded data to this variable.This works perfect when your data is comparatively small. If you have large data in response, the better way is to create file in Document directory and append all data to that file when connection receives data. Read this data after connection finishes loading.
This concept of saving data is applicable on android also.
I am writing an application in Ruby, which collect huge amount of data from API calls and stores it in a file. After that it processes it one by one. I was wondering if there is a way better than this to achieve the same?
Note: I want to process the records one by one by storing all of them locally because they may change during API calls.
I would look at storing the information in an in-memory key/value store (such as memcached or redis). If you use an in-memory key/value store, you can update information based on subsequent API calls rather than having multiple records in a file which represent the same data, just with different values.
Keep in mind, however, if your data is significantly large, you may run out of memory. That said, if you are into the gigabytes of data, the way you have implemented your solution may be the best route to take.
The problem:
I have been using for some time now my own cache system by using NSFileManager. Normally the data I receive is JSON and I just save the dictionary directly into cache (in the Documents Folder). When I need it back I will just go get it. I also implement, sometimes when I feel it's better, a NSDictionary on the root Folder with keys/values for the path for a given resource. For example:
Resource about weather in Geneve 17/02/2013, so I would have a key called GE_17_02_2013 and the value the path to the NSDictionary with the information.
Normally I don't need to do any complex queries. But, somehow, and from what I have been reading, when you have a lot of data, you should stick with Core Data. In my case, I normally have a lot of data, but I never actually felt the application going down, or suffering in terms of performance. So my questions are:
In this case, where sometimes (the weather thing was just an
example) I need to just remove all the data (a Twitter feed, for
example) and replace it by a completely new stream of data, is Core
Data worth? I think removing all the data, and inserting (populating) it, is heavier than just store the NSDictionary and replacing the old one.
Sometimes it would envolve images, textFiles, etc and the
NSFileManager does it perfectly, so what advantages could Core
Data bring in this cases?
P.S: I just saw this post, where this kind of question is made and numbers prove which one is actually faster. Still, I would like as well an empiric answer.
Core Data is worth using in every scenario you described. In fact, if an app stores more than preferences, you should probably use Core Data. Here are some reasons, among which, you'll find answers to your own problems:
is definitely faster than the filesystem, even if you wipe out everything and write it again as you describe (so you don't benefit to much from caching). This is basically because you can coalesce your operations and only access the store when needed. So if you read, write and read, you can save only once, the rest is done in memory, which is, needless to say, very fast.
everything is versioned and you can migrate from one version to another easily (while keeping the content the user has on the device)
80% of your model operations come free. Like, when something changes, you can override the willSave managed object method and notify your controllers.
using cascade makes it trivial to delete even very complex object structures
while is a bad idea to keep images in the database, you can still keep them on the filesystem and have core data delete them automatically when the managed object that represents them is deleted
is flexible, in fact is so flexible that you could migrate your project from using the local filesystem to using a server with very little modifications by writing a custom data store.
the core data designer basically creates the model objects for you. You don't need to create your own model classes (which you would have to if using the filesystem)
In this case ... is Core Data worth it?
Yes, to the extent that you need something more centrally managed than trying to draw up your own file-system schema. Core Data, and its lower-level cousin SQL, are still the best choice for persistence that we have right now. Besides, the performance hit of using NSKeyed(Un)Archiver to keep serializing/deserializing a dictionary over and over again becomes increasingly noticeable with larger datasets.
I think removing all the data, and inserting (populating) it, is heavier
than just store the NSDictionary and replacing the old one.
Absolutely, yes. But, you don't have to think about cache turnover like that. If you imagine Core Data as a static model, you can use a cache layer to ferry data in and out of the store. Need that resource about the weather? Check the cache layer. If it's not in there, make the cache run a fetch request. Need to turn over the whole cache? Have the cache empty itself then run a request to mark every entity with some kind of flag to show they are invalid. The expensive deletion you're worrying about can be done by a background process when you see that all your new data has been safely interned in the cache.
Sometimes it would envolve images, textFiles, etc and the
NSFileManager does it perfectly, so what advantages could Core Data
bring in this cases?
Unfortunately, not many. For blobs of data (which is essentially what Core Data does in these situations), storage and fetches to and from Core Data can quickly get costly. They can also take up a noticeably larger space on disk if they aren't compressed (which further decreases performance). If you need a faster alternative, use a store more suited to the task like Tokyo Cabinet or LevelDB, and use the entities in the Core Data store as a kind of stand-in that would, say, contain the key to the resource in one of those relational databases.
I read something about multi thread access on DB but still not sure best way to do that for reading/writing in conjunction with async network download.
For instance I will have a page with images from the web, so I'm retrieving them by their URLs, using AFNetworking but I want to check first on my DB and write on it (or disk) the retrieved image for future use.
What can be the best way to do that without blocking the UI (like when scrolling)?
If I do that with a singleton that reads/writes it block the main thread.
Thanks for any tips on that.
AFNetworking is not the tool for that. Instead, you can take advantage of the built-in functionality of NSURLCache--specifically Peter Steinberger's fork of SDURLCache.
In -applicationDidFinishLaunchingWithOptions:, do NSURLCache +setSharedCache: with an instance of SDURLCache (with some amount of disk space allocated). All requests made through UIWebView (and AFNetworking, for that matter) will automatically be routed through NSURLCache before the request is made to check the cache. It's unobtrusive, drop-in, and follows cache directives correctly, and it should solve your problem quite nicely.