I am working on an application where I have a connection to a database. The database contains from 300MB to 4GB worth of data as each customer has their own database. My issue that I am having is in gathering the data, because of the potential database size, just downloading and storing the information locally isn't possible. The data can get quite complex and can vary. For an example:
A customer has a Job and they want to search for that job from the app.
I then fetch a list of jobs matching the search criteria.
The customer sees the job they want to view and I start the gathering process.
This job can potentially touch many tables, sometimes repeatedly..
There is the jobs table, a relational table to map to a person. Then there is another table that contains non-customer relational information, then there are calendar events associated to the job, which in tun can associate different people. Then there are emails attached to the job, which in turn can bring in additional people and events.
So I have a working model that gathers all of this information. The problem I have is that I cannot figure out a great method of signaling to my view that the data is completely downloaded. My initial thought was to use the NotificationCenter to message when the certain parts of the task were finished, allowing the core Job object to notify the view when everything was complete.
I know this is a pretty generalized question, but I'm honestly stumped as to how to take an unknown number of table results and translate that into a notice that my app can actually use.
My initial recommendation would be Core Data. It's designed for this kind of problem. No, I'm not saying to download the entire database into Core Data. I'm saying to use Core Data to manage your object model, because that's what it's good at.
As you receive data from the server, compose it into NSManagedObjects and stick them in the data store. On the UI side, create an NSFetchedResultsController to keep you informed as the data updates asynchronously. You don't necessarily need to persist this store. You could just keep it in memory and throw it away whenever you're done with the query, but keeping it on disk could be a nice caching solution. Again, don't think of Core Data as "a local database." Think of it as a model persistence engine that you can query for objects.
One advantage of this model is that you can provide the best available data to the user as it becomes available. But say you really don't want to get the information until it's all available. That's fine, too. Just let the network side keep updating its context, and then only save it when everything's complete. That way NSFetchedResultsController gets a single atomic update. The nice things with Core Data is that it has these concepts built in, so you can adjust your update strategy without requiring massive redesign.
The Notification Center will work great for this.
Post the notification at logical points in your data load to trigger a UI update for your users.
Related
I understand that Core Data is essentially a self-contained local database, but I'm not sure if I should be using it in my app or not. Basically, it would be more for caching purposes if anything, since I retrieve all of my content from a web server database. Regardless, I was wondering if Core Data would be useful is any of these situations:
Scenario #1: I retrieve a list of "items" from the web server and feed them into a table view. This is essentially the first page the user sees. The table can be refreshed to retrieve more results, but existing items likely won't change. Over time this list of items could grow tremendously. Items can be deleted.
Scenario #2: A user has a friends list. This list of friends will stay the same unless he or she adds more friends. I imagine there will be a scenario where a friend deletes their account, in which case the friends list will be altered as well.
Scenario #3: Messages can be attached to items. They can't be edited or deleted, so the only change in state for a list of messages would be if a new message was added. Essentially the same as items, except they can't be deleted.
Actually, for your scenario I would say that you don't need any persistence in your app, but rather fetch your data from the server every time the app starts and just keep it in memory. There are a lot of apps which are doing it this way and this is totally fine behaviour.
However, there are some drawbacks of not using persistence:
worse offline experience for your user since they depend on a network connection, so effectively without a connection they can't do anything within your app
risk of slow loading
On the plus side we have:
using Core Data in your app is a huge implementation overhead (especially if you haven't used it before)
after having integrated Core Data, you still have a lot of issues to tackle, first and foremost: data synching between your app and the backend
If you decide to go for persistence, also take a look at alternatives to Core Data like Realm.
Finally, my advice still is to not use Core Data in your situation. However, keep in mind that you can build a version of your app that doesn't use persistence. And then, once you see that your app is well-received and gets more attention, you can still go and add persistence later on.
I am somewhat new to Core Data and have a general question.
In my current project, users can access data reported by various sensors in each county of my state. Each sensor is represented in a table view which gathers its data from a web service call. Calling the web service could take some time since this app may be used in rural areas with slow wireless connectivity. Furthermore, users will typically only need data from one or two of the state's 55 counties. Each county could have anywhere from 15 to 500 items returned by the web service. Since the sensor names and locations change rarely, I would like the app to cache the data from the web service call to make gathering the list of sensors locations faster (and offer a refresh button for cases where something has changed). The app already uses Core Data to store bookmarked sensor locations, so it is already set up in the app.
My issue is whether to use Core Data to cache the list of sensors, or to use a SqlLite data store. Since there js already a data model in place, I could simply add another entity to the model. However, I am concerned about whether this would introduce unnecessary overhead, or maybe none at all.
Being new to Core Data, it appears that all that is really happening is that objects are serialized and their properties added as fields in a SqlLite DB managed by Core Data. If this is the case, it seems there really would not be any overhead from using the Core Data store already in place.
Can anyone help clear this up for me? Thanks!
it appears that all that is really happening is that objects are
serialized and their properties added as fields in a SqlLite DB
managed by Core Data
You are right about that. Core Data does a lot more, but that's the basic functionality (if you tell it to use a SQLite store, which is what most people do).
As for the number of records you want to store in Core Data, that shouldn't be a problem. I'm working on a Core Data App right now that also stores over 20,000 records in Core Data and I still get very fast fetch times e.g. for auto completion while typing.
Core Data definitely adds some overhead, but if you only have few entites and relationships and are not creating/modifying objects in more than one context, it is negligible.
Being new to Core Data, it appears that all that is really happening is that objects are serialized and their properties added as fields in a SqlLite DB managed by Core Data. If this is the case, it seems there really would not be any overhead from using the Core Data store already in place.
That's not always the case. Core Data hides its storage implementation from the developer. It is sometimes a SQL db, but in other cases it can be a different data storage. If you need a comprehensive guide to CoreData, I recommend this objc.io article.
As #CouchDeveloper noted, CoreData is a disk io/CPU bound process. If you notice performance hits, throw it in a background thread (yes - this is a pretty big headache), but it will always be faster than the average network.
Here's and example of what I am talking about:
Take Twitter for iOS. Whenever you tweet, the tweet is sent to the database, and then it is also displayed on your device as part of the list of tweets.
How is the list of tweets that you see on your device updated after just sending one tweet? Here are some possible ways that I thought of how it could be done, but what Im asking for is which one is the best method of doing so:
The whole list of recent Tweets is re-downloaded from the remote Twitter server after sending a tweet (I highly doubt this, as this would take a relatively long time, when it really is just appending one Tweet to the array of Tweets displayed)
The local array that holds the Tweet objects is updated separately from the database (For example, it updates the database, and then updates its array with the same data you sent to the database, and never downloads the Tweet you just sent since you don't need to, because you already have it locally, since you composed it)
Is Core Data capable of updating the remote data server AND the array all in one (or relatively few) step(s)? (Sorry, if this is the obvious answer and if it sounds like I didn't look into it, but I did read about Core Data and started a tutorial. Its just that there is so much content that it would take me a whole day or two just to figure out if its appropriate for my application)
Is there an alternative way of managing this?
Also, if its one of the latter two ideas above, are you able to update the table view cells by just updating the local array and reloading the cells from that array without loading your one tweet from the database? I'm just curious about what would be the most efficient way of doing this.
So again, my main question reworded is: how do you keep data that you sent to a remote database and the local data (stored in a mutable array) in sync whenever you do a tiny single update (such as sending a Tweet) without having to reload all of the data from the database (when there is other content [i.e. other Tweets]) already loaded.
(I am aware that no one except Twitter developers know exactly how Twitter actually done, but I'm just using this Twitter functionality as an example. This same concept could be applied to any similar app.)
(Also, this is a conceptual question about dataflow, so I don't need to see any code, but suggestions to use different technologies like Core Data, or just updating an array will be appreciated.)
(I've been looking into this, and all the different ways of doing it, and it is becoming very time consuming, so I figured to ask you guys who have experience. Additionally, this could help someone else who has similar questions.)
(Sorry if it looks like I'm asking a bunch of questions, but I'm basically asking the same question in different ways, and offering possible solutions.)
Any insight is appreciated!
Immutable messages like tweets are actually quite easy to handle -- server side, and in your app.
When you send a tweet from your client to the server, you also update your "main context" (see "Managed Object Context") which in turn sends notifications to your controller (see NSFetchedResultsController which in turns updates your table view according your local model residing in the Managed Object Context.
Updating from the server is just merging the local tweets with the new ones added in the meantime.
Since there is no mutable tweet, synchronization is really no big deal. As mentioned in the comment, if there were mutable tweets (or any kind of messages) the synchronization will become much more complex.
Core Data will NOT automatically update a remote server. But there are solutions to "view" a remote database through Core Data - see NSIncrementalStore and a related third party libraries (AFIncrementalStore).
This is ridiculously trivial. You update your local database and send off the remote update at the same time.
You use the remote response to mark your local record as synched or try updating again later.
I have what I would presume is a very common situation, but as I'm new to iOS programming, I'm not sure of the optimal way to code it.
Synopsis:
I have data on a server which can be retrieved by the iPhone app via a REST service. On the server side, the data is objects with a foreign key (an integer id number).
I'm storing the data retrieved via REST in Core Data. The managed objects have an "objId" attribute so that I can uniquely identify the managed objects in the rest of my code.
My app must always reflect the server data.
On subsequent requests made to the server:
some objects may not be returned, they have been deleted on the server - in which case I need to delete the corresponding objects from Core Data - so that I'm reflecting the state of the server correctly.
some objects have attributes which have changed, therefore the corresponding managed objects need updating with the new data.
my solution - and question to you
To get things going in my app, I made the easiest solution of deleting all objects in Core Data, then adding all new objects in, created with the latest server side data.
I don't think this is the best way to approach it :) As I progress on with my app, I now want to link up my tableview with NSFetchedResultsController, and have realised that my approach of deleting everything and re-adding is not going to work any more.
What is the tried and trusted way of syncing Core Data with server side data?
Do I need to make a fetch request for each object id I get back from the server, and then update the object with the new data?
And then go through all of the objects in core data and see which ones have not been updated, and delete those?
Is that the best way to do it? It just seems a little expensive to do a fetch for each object in Core Data, that's all.
Pseudo code is fine for any answers :)
thanks in advance!
Well, consider your download. First, you should be doing this in a background thread (if not, there are lots of SO posts that talk about how to do that).
I would suggest that you implement what makes sense first, and then, after you can get valid performance data from running Instruments, consider performance optimization. Of course, use some common sense on "easy" performance stuff (your design can take care of the big ones easily enough).
Anyway, get your data from the online resource, and then, for each object fetched, use the "unique object id" to fetch the object from core data. You know there is only one object with that ID, so you can set fetchLimit to 1 on your fetch request. You can also configure your "object id" attribute to be an INDEX in the database. This way, you get the fastest search from the underlying database, and it knows to stop looking once it finds your one object. This should be pretty snappy.
Now you have your object. Change any attributes necessary. Save, rinse, and repeat.
Furthermore, for several reasons, you may want to know when objects were last updated. I'd suggest adding a timestamp to each object that gets changed with the current time every time an object is changed. This will also help in deleting objects. Since your online database does not tell you which objects are deleted, you must have some way to know that an item is "old and no longer needed."
An easy way to do this is to remember the time you started your update. After processing all objects from the download, you now have a way to find all the objects that were deleted from the online database. Basically, any object with a "last update" timestamp before the time you began the update should be removed (since they were not added or modified in the last update). You can also index the database on this field, which will make finding those objects faster - unless your database is huge, I'd wait to see what Instruments has to say about this one though.
I am currently creating an iOS app, which connects to a database and asynchronously downloads a JSON object of data to display in a table view.
As it currently stands, this is an ok way to do it. However, when the database starts getting much larger, this will cause a massive inconvenience. I'm reasonably proficient in Objective-C but not so much in the database side of things. What would be the best way to get this data from the server, and keep it in the app? At the moment, I have a custom class object storing the data for each of the 'objects' in the JSON object. There will however be many other aspects of the app that the database will handle, such as invites, logins and user details.
Would core data be the way to go? I.e duplicating the database (to a certain extent) and storing it locally, then accessing from there. As I said, i'm not really sure which route to take here, so any advice would be real appreciated.
Core location is for handling location (satellite (and wifi) positionning).
I guess you mean Core Data. Core Data is a graph object model which allows you to manipulate data as objects. You don't dig directly into the database, you ask for objects instanciation through predicates (kind of where clause in SQL) and the manipulate the objects.
This stated, it all depends on what is a "big" database. If it's really big you could consider copying locally a part of it and ask for what's remaining from the server through your webservice.
Another question that you could ask yourself is the quantity of data that never change and if your website database and your app database needs to get synchronized (if your website database is always changing then it would be dumb to copy it in your app totally and always synced your app..).
Links :
Introduction to Core Data
Difference between Core Data and a Database (Cocoa With Love)
edit :
A question you can ask yourself is where your data needs to be saved ?
if your app is just for printing 20 cells out of a total of 200 cells then i would go for a total download of your 200 cells. The load of the other cells will be with no delay after first download, especially appreciated if you're using table view cells with reusable cells
is a delay of some seconds acceptable between the 20 first cells and the 20 following ? I think there is no real "good" answer to your question, it depends on many factors (purpose of your app, acceptable time between loads, does the info needs to be modified and saved back to server or locally, what kind of customers, what your app will do with the cells, if you have a database locally will it be totally independant from "mother" database (if no, what kind of synchronization), etc.)
Trying to sum up things according to what I've understood of your needs, I would say that webservices is good if you just need to retrieve info and exploiting it after without saving it back (even if you can do it actually having services allowing you to do it), having a database locally is good if you need your app to be independant from your server in some ways.
Only you has the key to answer all this and take a decision according to your needs and your knowledge of your application and your customers.
Something like JSON or SOAP is the way to go with getting structured data from a web service into objects in your iPhone app.
Storing relational data on the iPhone itself is easy with SQLite. Here's a decent looking tutorial.
Make things easy for yourself by writing a data layer, abstracting away calls to the database, to avoid dotting SQL queries all over your code in places it shouldn't be, like the UI.