My situation:
I have an app that needs to store 10,000 - 30,000 locations in some sort of storage method, which are then displayed on a MKMapView as individual pins. I also have a server that needs to be able to add to the database through pushing out changes.
Through grouping pins I've eliminated all issues with the MKMapView, my biggest focus is now on speed, storage and being able to add and replace the storage contents. What I'm currently doing is I have a text file of currently 1,000 locations as JSON-formatted, then they're just read as an array and sent to my custom map view (no issues there). My only issue is how I could update that text file (rather than downloading massive amounts of data), and store almost 30,000 locations.
Is this even feasible? It seems my current setup could scale pretty much perfectly, it's just this updating system that is causing me a headache.
Your current setup won't scale forever because you have to load the entire file into memory in one chunk. Eventually it will get to large to manage and will eat up to much memory. Unable to purge memory in the event of system low-memory, the system will shut your app down i.e. it won't be able to stay in the background but will have to reboot each time the user switches back to it.
To update, you will have to load in the entire file, parse the JSON, figure out how to update the resulting data structure, then write it all to file. One error anywhere in the process could corrupt the entire file.
You really need to look at using Core Data or even SQL. Core Data has a learning curve but once you master it, it makes implementing designs like your app trivial. You also get automatic scaling and efficient memory management.
Related
My question surrounds around one single point - data management in mobile application. I have created a mobile application where data comes from server. The data includes both text and images. Following are the steps I am doing for this :
First launch :
1. Get server data.
2. Save server data in Sqlite database.
3. Show Sqlite data.
Next launches :
1. Show Sqlite data.
2. Get server data in background.
3. Delete previous Sqlite data.
4. Save new server data in Sqlite database.
5. Show Sqlite data.
I have couple of questions on these steps :
1. Is this the right approach ? Other way could be showing data every time from server but that would not display the data on screen immediately (depending on internet speed).
2. I also thought of comparing the Sqlite data with the new server data. But faced a big challenge. The new server data might have new records or deleted records. Also, I could not find an appropriate approach to compare each database field with JSON data.
So what is the best approach to compare local Sqlite data with new server data ?
3. Each time I delete the Sqlite data and insert new data and then refresh the screen (which has a UITableView), it blinks for a second which is obvious. How to avoid this issue if steps 3, 4, 5 are followed ?
4. How should I proceed with data update in case I come back on the screen each time or when the application becomes active ? I am very aware of NSOperationQueues or using GCD for that matter. But what if I am crazy and go back and forth to screen again and again. There will be a number of NSOperations in the queue.
It's a challenge to synchronise server data, I've done that before, and if you can spend time on it I'd say it's the best solution.
You may need creation and modification dates on both server and local objects, to compare them - this will let you decide which objects to add, update and delete.
If the server sends you only the recently updated objects you can save a lot of traffic and improve performance (but deleted objects will be harder to detect).
If the data is only changed in the server it's easier, when the app can change the data too it becomes more complicated (but it seems that it's not your case). It also depends on how complex the database is, of course.
If you don't want to invest some time in doing this, just fetching all data everytime works too, even if it is not ideal! Instead of showing the old data and blinking it, you can just make the user wait 2-3 seconds when entering, while you get the new data. Or instead you can fetch the data only when starting the app, and so when you get to that view controller it will be ready already.
It's a complex problem that everyone faces at some point, so I'm curious to see what other people will suggest :)
This is a good question.
I personally think downloading data, store locally and later try to sync is a dangerous scenario. Easy to introduce bugs, master <-> slave issues (what data should be master, if multiple devices would be used etc.)
I think something like this could be a working approach:
1. I would try to look at possibilities to lazy load the data from the server on-demand. That is when a user have a View that should display data, load that specific data with the creation of that specific View. This ensures the data is allways in sync.
2. Tackling the need to reload data from server from every view, could be done by simply storing the downloaded data as objects in memory (not using SqlLite). The view will try to load the needed data trough your cache manager, and it would serve it from memory, if available. If not in memory simply get the data from your server and add it to your memory cache.
The memory cache could be a home made data manager wrapping a Dictionary stored on you AppDelegate, or some global "Singelton" to wrap the cache management/storing and data loading.
3. With lazy loaded data and memory cache you would need to make sure any updates (changes, new records, deleted records) updates your memory data model, as well as pushing these changes to the server as soon as possible. Depending on data size etc. you could force the user to wait, or do it directly as background process.
4. To ensure the data is in sync, you should make sure that you periodically invalidate (delete) the local memory records in the cache and thereby force data updates from the server. Best approach would probably be to have a last updated timestamp for each record in the memory cache. So the periodical invalidator would only delete "old records" from the memory cache (once again not from the server).
To save server from unnecessary data load, the data should still load on demand when the user needs it in a view, and not as part of "cache invalidation".
5. Depending on the data size you might need to look at "cache invalidation". Could be as simple as when xx records are stored, start deleting old objects from memory cache (not server, only locally on device).
6. If data sync is absolutely critical you might want to look at refreshing your memory cache for a record, just before you allow the user to change data. E.g. when user taps "Edit" or similar, you grab the latest data from server for that record. This is just to make sure the user is not going to update a record using outdated data and thereby accidentally overriding any changes made remote, or on another device etc.
--
My take on it. I do not believe there is a "perfect right way" to do this. But this would be what I would try to do.
Hope this will help with some ideas and inspiration.
How about this:
If data exists in SqlLite, load into "in-memory" copy and show it
In background load new server data
delete old sqlite data if it exists (note that the in-memory copy remains)
save new server data to sqlite
load new sqlite data into "in-memory" copy and show it.
If no data was found in step 1, display a "loading" screen to the user during step 2.
I'm making the assumption that the data from SqlLite is small enough to keep a copy in memory to show in your UITable view (The UITable view would always show data from in-memory).
It may be possible to combine steps 4 and 5 if the data is small enough to hold two copies in memory at the same time (you would create a new in-memory copy and swap with the visible copy when complete).
Note:
I don't talk about error handling here, but I would suggest that you don't delete the sqlite data until you have new data to replace it with.
This approach also eliminates the need to determine if this is the first launch or not. The logic always remains the same which should make it a little easier to implement.
Hope this is useful.
You can do same things more efficiently by MultiVersion Concurrency Control (MVCC), which uses a counter (sort of a very simple "time stamp") for every data record, which is updated whenever the record is changed means you need to get those data which is Updated after last sync call that reduces lots of redundant data and bandwidth.
Source: MultiVersion Concurrency Control
So this is more of a conceptual architecture question. I'm making a messaging app in iOS. Devices can send image messages. I am saving custom Message objects (which include images) to disk using NSCoder protocol. I want to remove the messages from memory when I don't need them (i.e. when the user logs out and disappears from users list view). I am appropriately encoding that user's messages to disk, and loading them when the user re-appears. However, on the messaging view (a dynamic UITableView where each cell displays the message content), the image content of the messages are cached within the UIImageView subview.
So, I am essentially creating duplicates of the images when I load the messages from disk. Every time a user logs out and then in (i.e. disappearing and then reappearing), the associated messages are saved and then loaded from disk (recreated objects), and my memory usage creeps upward once I go to the private message view and scroll all the way up in order to display all messages.
Ultimately, I want to clean the cache of message images so that I can free up memory when a user is no longer around. I have a good reason for saving messages locally; my question is: what is the best design for saving messages/images to disk, removing them from cache memory, then reloading them when I need to populate a dynamic UITableView?
I'd advice you to use some ready caching library like: SDWebImage, FastImageCache or Haneke. All of them are both using disc and memory cache, what will improve the speed of your tableView and decrease number of download requests.
You can of course make your custom solution, but it won't work as well as something written during months and tested via many developers. However if you want to, I'd suggest to take a look into one of these libraries and start by considering what they've done to make everything 'speedy'.
This way you just save URL in your Message instead of full image. These libraries will check URL, understand that they already have the image in memory cache and load it. You don't have to store it in your array of objects nor download again or save by yourself on disc.
Keeping your message objects on the disc seems weird, as this is slow - better to use database - CoreData/Mantle/FBDB etc, however that's your decision :) Also storing on the disc makes your Messages very easy to reverse engineering attacks and isn't safe.
I'm building an app that makes extensive use of CoreData and a lot of my models have UIImage and NSData properties (for images and videos). Since it's not a great idea to store that data directly into CoreData, I built a file manager class that writes the files into different buckets in the documents directory depends on the context in which was created and media type.
My question now is how do I manage the documents directory? Is there a way to detect how much space the app has used up out of its total allocated space? Additionally, what is the best way to go about cleaning those directories; do I check every time a file is written or only on app launch, ect ect.
Is there a way to detect how much space the app has used up out of its total allocated space?
Apps don't have a limit on total allocated space, they're limited by the amount of space on the device. You can find out how much space you're using for these files by using NSFileManager to scan the directories. There are several methods that do this in different ways-- check out enumeratorAtPath:, for example. For each file, use a method like attributesOfItemAtPath:error: to get the file size.
Better would be to track the file sizes as you create and delete files. Keep a running total, stored in user defaults. When you create a new file, increase it by the amount of new data. When you remove a file, decrease the running total.
Additionally, what is the best way to go about cleaning those directories; do I check every time a file is written or only on app launch, ect ect.
If these files are local data that's inherently part of the associated Core Data object, the sensible approach is to delete a file when its Core Data object is deleted. The managed object needs the data file, so don't delete the file if you still use the object. That means there must be some way to link the two, but I'm assuming that's already true since you say that these files are used by managed objects somehow.
If the files are something like cached data that's easily re-created or re-downloaded, you should put them in the location returned by NSTemporaryDirectory(). Then iOS can delete them when it thinks the space is needed. You can also clear out old files whenever it seems appropriate, by scanning for older files or ones that haven't been used in a while (the details depend on exactly how you use the files).
I'm making a catalog where the cells in my collection view will be either an image with a label or a pdf. There will be many collections and they themselves will be static. I want the user to be able to save the cells he likes and view them in his own custom view.
1) I could to store the image as data in Core Data.
2) I could just include the image in my App Bundle and load the image from there every time my app starts.
I've got it into to my head that reading data from a Core Data Store would give me more options when building my app as well as offer some boost in performance as opposed to reading it from the app bundle. Is that true? Keeping in mind of course that most of the data is static.
It seems inefficient to have images both serialized images in my app bundle and the pure data as well.
I think I'd rather have it all in the store but they have to be loaded from the bundle at some point in code right?
I'd love to know how other developers do it.
Now in Core Data there is an "allows external storage" option for binary data, which basically means if your file is bigger than 1 MB it will be stored automatically outside of your database, and you have to do nothing differently. In my opinion that's the way to get the best of both worlds, increased performance + automatization + fast queries (although they are slower than usual when you allow external storage, but still faster than doing it yourself)
I am currently creating an iOS app, which connects to a database and asynchronously downloads a JSON object of data to display in a table view.
As it currently stands, this is an ok way to do it. However, when the database starts getting much larger, this will cause a massive inconvenience. I'm reasonably proficient in Objective-C but not so much in the database side of things. What would be the best way to get this data from the server, and keep it in the app? At the moment, I have a custom class object storing the data for each of the 'objects' in the JSON object. There will however be many other aspects of the app that the database will handle, such as invites, logins and user details.
Would core data be the way to go? I.e duplicating the database (to a certain extent) and storing it locally, then accessing from there. As I said, i'm not really sure which route to take here, so any advice would be real appreciated.
Core location is for handling location (satellite (and wifi) positionning).
I guess you mean Core Data. Core Data is a graph object model which allows you to manipulate data as objects. You don't dig directly into the database, you ask for objects instanciation through predicates (kind of where clause in SQL) and the manipulate the objects.
This stated, it all depends on what is a "big" database. If it's really big you could consider copying locally a part of it and ask for what's remaining from the server through your webservice.
Another question that you could ask yourself is the quantity of data that never change and if your website database and your app database needs to get synchronized (if your website database is always changing then it would be dumb to copy it in your app totally and always synced your app..).
Links :
Introduction to Core Data
Difference between Core Data and a Database (Cocoa With Love)
edit :
A question you can ask yourself is where your data needs to be saved ?
if your app is just for printing 20 cells out of a total of 200 cells then i would go for a total download of your 200 cells. The load of the other cells will be with no delay after first download, especially appreciated if you're using table view cells with reusable cells
is a delay of some seconds acceptable between the 20 first cells and the 20 following ? I think there is no real "good" answer to your question, it depends on many factors (purpose of your app, acceptable time between loads, does the info needs to be modified and saved back to server or locally, what kind of customers, what your app will do with the cells, if you have a database locally will it be totally independant from "mother" database (if no, what kind of synchronization), etc.)
Trying to sum up things according to what I've understood of your needs, I would say that webservices is good if you just need to retrieve info and exploiting it after without saving it back (even if you can do it actually having services allowing you to do it), having a database locally is good if you need your app to be independant from your server in some ways.
Only you has the key to answer all this and take a decision according to your needs and your knowledge of your application and your customers.
Something like JSON or SOAP is the way to go with getting structured data from a web service into objects in your iPhone app.
Storing relational data on the iPhone itself is easy with SQLite. Here's a decent looking tutorial.
Make things easy for yourself by writing a data layer, abstracting away calls to the database, to avoid dotting SQL queries all over your code in places it shouldn't be, like the UI.