We have a product that uses central CouchDB databases per client replicating to Apps running on user's iPads. Most of the database can replicate normally but we have two categories of document that we want to filter:
Documents with an owner - we want to filter the replication to only the current users documents (and documents with no specified owner).
Last X documents of some type. For some sorts of documents we only want to leave the last 10 (say) copies on the iPad.
We can set up both rules easily enough using filtered replication - so that the server only presents the subset of documents we want to the iPad for replication. Except... it does not work.
If a document has no owner (replicated) and later an owner is specified, it vanishes from the replication stream - but not from the iPad. In fact the version of the document that remains on the iPad still has NO owner and so we cant even hide it in code.
When a document becomes the 11th oldest and vanishes from the replication stream, it does not vanish from the iPad. Indeed unless the iPad database is rebuilt all versions of these documents end up there, and no longer replicate which is worse than just replicating them all in the first place.
We did find a hacky workaround - in the case where a document gains a new owner OR becomes older than X, we duplicate it and delete the original. The delete propogates to the iPad and the new document is filtered out of replication. This worked well enough (although it is a bit inefficient). However then we realised the newly copied document had lost all of its revision information and we were relying on the revisions to track changes!
So - does anyone have any other suggestion? What we are looking for is a mechanism to pull a document from the iPad replicas on demand. I am aware we could instruct the iPad to delete the documents locally - but then sooner or later those deletes would leak back to the server and destroy the original?
... we were relying on the revisions to track changes
IMHO this is the most interesting point to talk about an alternative solution.
I'm sorry but i have to say you using the CouchDB revision control in the way it that is not recommended. The document revisions are temporary. The best way to track changes of a document is to write a changes log inside or outside the doc.
How would you persist changes outside the doc itself - yes, you would create new docs. Surprise: you "Hack" is the right solution \o/
Maybe you shaking your head and your are not happy because you have tried to remove docs from the iPad to make them invisible client-side. That was the starting point of your "Hack", right?
My recommendation is to not combine "visibility" and "existence". Better would be to use your know-how with building view-indexes server-side in the same way client-side with PouchDB. Let the replication just handle replication - thats hard enough. Use views/filters client- and server-side to solve visibility requirements.
Related
I am developing a news application . I am using NSCacheDirectory to save to image's . I want to know when should I delete the images from the cash . I know I could probably delete it when I receive a memory warning (or it would be automatically deleted in a low memory condition) but I don't understand why to keep the images for so long and delete them only if I receive the memory warning . I also tried to check if all the images saved in NSCashDirectory are in the data which I get (from the internet) and if not then delete it , but I am sure performance would be a issue . Is there a way to clear cash after a specific time interval
P.S. I am quite new to iOS programming .
Since you are developing a news application I am going to assume that the news will be updated at certain intervals. This means that you could probably define a lifetime for an article stored on the device. So what you could do is say every day, mark all articles older than seven days as trashable. Then, when it's safe to do so (App launch or when user is reading new content) you can delete said content.
I don't know your audience but traditionally it's safe to assume that people won't read the same article multiple times so you can definitely make your window for stale content quite small.
That being said if you implement a bookmarking facility you could approach this in a similar fashion too. You could for instance assume that the user will most likely interact with the most recently bookmarked articles. Remember, in this recency is defined by the user not the article date. Articles bookmarked most recently by the user will possibly be read or shared more likely than older ones.
You could also have an option in your setting where the user can choose to delete all cached content. Since your content should be living on a server it shouldn't be impossible for the user to fetch old content.
As far as you are concerned each article could be broken up into an article mapping and an article. An article mapping should contain the bare minimum of information to allow for the display of content in an article feed (things like headline, thumbnail URL etc). Where your article would contain all the content. When the user requests to delete all cached data you could delete all the images and keep all the mappings (at least the most recent ones) on the device still.
Happy coding
There has been a lot of discussion lately about the issues with iCloud and Core Data and how Apple's APIs are currently broken in iOS 5 and possibly iOS 6.
Is it possible, given the current state of Apple's Core Data API, to reliably sync across multiple devices using iCloud?
If so, how would you do this? If not, please recommend an alternative approach.
This blog post will lead you to a chain of recent articles about the travails of developers attempting this approach.
From my own understanding and experience, I believe it is doable, but don't buy into the idea that you will get anything "for free". Depending on your data model, you may be better off syncing your whole persistent store as a document rather than using the documented core data / iCloud approach.
You may have better luck if you're already comfortable with Core Data. Just be sure you think through how to handle several important cases.
One is what to do if the user signs out of their iCloud account. When this happens, the local ubiquitous persistent store is deleted. If it makes sense for the user to still have access to their data, you'll need to manage a copy in local storage, and then manage resynchronizing when they sign back in.
Another is that changes can apparently be quite slow to propagate by default, so you may want to consider an alternative mechanism, such as the key value store, to quickly propagate sufficient information to avoid a bad user experience.
Conflict management is perhaps the most challenging (depending on your model). While the framework provides a mechanism to inform you of conflicts, you are on your own for providing a mechanism to resolve them, and there are reports that the conflict notifications may not be reliable (see linked articles), which seems strongly linked to the lag in updating.
In short, if you go into this understanding that the actual support is pretty bare bones and that you'll need to code very defensively, you may have a chance. There aren't any good recipes out there, so if you do make it work, please come back and tell us what works!
It depends on what you want to do. There are two types of Core Data-iCloud integration, as described here: http://developer.apple.com/library/ios/#releasenotes/DataManagement/RN-iCloudCoreData/_index.html
There are broadly speaking two types of Core Data-based application that integrate with iCloud:
Library-style applications, where the application usually has a single persistent store, and data from the store is used throughout the application.
Examples of this style of application are Music and Photos.
Document-based applications, where different documents may be opened at different times during the lifetime of the application.
Examples of this style of application are Keynote and Numbers.
If you're using the library-type, this article is the first of a series that goes into a lot of the problems that will come up: http://mentalfaculty.tumblr.com/post/23163747823/under-the-sheets-with-icloud-and-core-data-the-basics.
You can also check out sessions 218 (for document-based) or 227 (for library-style) of this year's wwdc.
As of iOS 7, the best solution is probably the Ensembles framework: https://github.com/drewmccormack/ensembles
Additionally, there is a promising project which will essentially allow you to do the same thing using a different cloud service.
Here is a link to the repository: https://github.com/nothirst/TICoreDataSync
Project description:
TICoreDataSync is a collection of classes to enable synchronization via the Cloud (including Dropbox) of Core Data-based applications (including document-based apps) between any number of clients running under Mac OS X or iOS. It's designed to be easy to extend if you need to synchronize via an option that isn't already supported.
Reasons for why iCloud is not currently reliable:
"Sometimes, iCloud simply fails to move data from one computer to another."
"Corrupted baselines are [a] common obstacle.... There is no recovery from a corrupted baseline, short of digging in to the innards of your local iCloud storage and scraping everything out, and there is no visible indication that corruption has occurred — syncing simply stops."
"Sometimes, when initializing the iCloud application subsystem, it will simply return an opaque internal error. When it fails, there’s no option to recover — all you can do is try again (and again…) until it finally works."
"[W]hen you turn off the “Documents & Data” syncing option in the iCloud system preferences, the iCloud system deletes all of your locally stored iCloud data[.]"
When you sign out of iCloud, the system moves your iCloud data to a location outside of your application’s sandbox container, and the application can no longer use it.
"In some circumstances (and we haven’t been able to figure out which, yet), iCloud actually changes the object class of an item when synchronizing it. Loosely described, the object class determines the type of the object in the database[.]"
"In some cases (again, not all the time), iCloud may do one of the following:
Owner relationships in an item’s data will point to the wrong owner;
Owner items get lost in synchronization and never appear on computers other than the one on which they were created. (This leads to the item never appearing in the UI on any other machine.) When this happens, bogus relationships get created between blob items and an arbitrary unrelated owner."
"Sometimes (without any apparent consistency or repeatability), the associated data for an object (for example, the PDF data for a PDF item, or the web archive data for a Web Archive item) would simply fail to show up on the destination machine. Sometimes it would arrive later (much later — minutes or hours)."
Quoted and paraphrased from these sources:
http://www.imore.com/debug-12-icloud-core-data-sync
http://rms2.tumblr.com/post/46505165521/the-gathering-storm-our-travails-with-icloud-sync
Note: I have seen one article where the author mentions getting it to work for iOS 6+, but they don't provide any examples: http://zaal.tumblr.com/post/46718877130/why-you-want-to-use-core-data-icloud-sync-if-only-it
As a reference, here are Apple's docs on iCloud + Core Data:
http://developer.apple.com/library/ios/#releasenotes/DataManagement/RN-iCloudCoreData/
http://developer.apple.com/library/ios/#documentation/General/Conceptual/iCloudDesignGuide/Chapters/DesignForCoreDataIniCloud.html
http://developer.apple.com/library/ios/#documentation/cocoa/Conceptual/CoreDataVersioning/vmCloud/vmCloud.html
And here is an example app:
http://developer.apple.com/library/ios/#DOCUMENTATION/General/Conceptual/iCloud101/Introduction/Introduction.html
The Apple developer tutorial on using the iCloud API to manipulate documents might be a good place to start.
Your Third iOS App introduces you to the iCloud document storage APIs. You use these APIs to store and manipulate files in a user’s iCloud storage.
There has been a lot of discussion lately about the issues with iCloud and Core Data and how Apple's APIs are currently broken in iOS 5 and possibly iOS 6.
Is it possible, given the current state of Apple's Core Data API, to reliably sync across multiple devices using iCloud?
If so, how would you do this? If not, please recommend an alternative approach.
This blog post will lead you to a chain of recent articles about the travails of developers attempting this approach.
From my own understanding and experience, I believe it is doable, but don't buy into the idea that you will get anything "for free". Depending on your data model, you may be better off syncing your whole persistent store as a document rather than using the documented core data / iCloud approach.
You may have better luck if you're already comfortable with Core Data. Just be sure you think through how to handle several important cases.
One is what to do if the user signs out of their iCloud account. When this happens, the local ubiquitous persistent store is deleted. If it makes sense for the user to still have access to their data, you'll need to manage a copy in local storage, and then manage resynchronizing when they sign back in.
Another is that changes can apparently be quite slow to propagate by default, so you may want to consider an alternative mechanism, such as the key value store, to quickly propagate sufficient information to avoid a bad user experience.
Conflict management is perhaps the most challenging (depending on your model). While the framework provides a mechanism to inform you of conflicts, you are on your own for providing a mechanism to resolve them, and there are reports that the conflict notifications may not be reliable (see linked articles), which seems strongly linked to the lag in updating.
In short, if you go into this understanding that the actual support is pretty bare bones and that you'll need to code very defensively, you may have a chance. There aren't any good recipes out there, so if you do make it work, please come back and tell us what works!
It depends on what you want to do. There are two types of Core Data-iCloud integration, as described here: http://developer.apple.com/library/ios/#releasenotes/DataManagement/RN-iCloudCoreData/_index.html
There are broadly speaking two types of Core Data-based application that integrate with iCloud:
Library-style applications, where the application usually has a single persistent store, and data from the store is used throughout the application.
Examples of this style of application are Music and Photos.
Document-based applications, where different documents may be opened at different times during the lifetime of the application.
Examples of this style of application are Keynote and Numbers.
If you're using the library-type, this article is the first of a series that goes into a lot of the problems that will come up: http://mentalfaculty.tumblr.com/post/23163747823/under-the-sheets-with-icloud-and-core-data-the-basics.
You can also check out sessions 218 (for document-based) or 227 (for library-style) of this year's wwdc.
As of iOS 7, the best solution is probably the Ensembles framework: https://github.com/drewmccormack/ensembles
Additionally, there is a promising project which will essentially allow you to do the same thing using a different cloud service.
Here is a link to the repository: https://github.com/nothirst/TICoreDataSync
Project description:
TICoreDataSync is a collection of classes to enable synchronization via the Cloud (including Dropbox) of Core Data-based applications (including document-based apps) between any number of clients running under Mac OS X or iOS. It's designed to be easy to extend if you need to synchronize via an option that isn't already supported.
Reasons for why iCloud is not currently reliable:
"Sometimes, iCloud simply fails to move data from one computer to another."
"Corrupted baselines are [a] common obstacle.... There is no recovery from a corrupted baseline, short of digging in to the innards of your local iCloud storage and scraping everything out, and there is no visible indication that corruption has occurred — syncing simply stops."
"Sometimes, when initializing the iCloud application subsystem, it will simply return an opaque internal error. When it fails, there’s no option to recover — all you can do is try again (and again…) until it finally works."
"[W]hen you turn off the “Documents & Data” syncing option in the iCloud system preferences, the iCloud system deletes all of your locally stored iCloud data[.]"
When you sign out of iCloud, the system moves your iCloud data to a location outside of your application’s sandbox container, and the application can no longer use it.
"In some circumstances (and we haven’t been able to figure out which, yet), iCloud actually changes the object class of an item when synchronizing it. Loosely described, the object class determines the type of the object in the database[.]"
"In some cases (again, not all the time), iCloud may do one of the following:
Owner relationships in an item’s data will point to the wrong owner;
Owner items get lost in synchronization and never appear on computers other than the one on which they were created. (This leads to the item never appearing in the UI on any other machine.) When this happens, bogus relationships get created between blob items and an arbitrary unrelated owner."
"Sometimes (without any apparent consistency or repeatability), the associated data for an object (for example, the PDF data for a PDF item, or the web archive data for a Web Archive item) would simply fail to show up on the destination machine. Sometimes it would arrive later (much later — minutes or hours)."
Quoted and paraphrased from these sources:
http://www.imore.com/debug-12-icloud-core-data-sync
http://rms2.tumblr.com/post/46505165521/the-gathering-storm-our-travails-with-icloud-sync
Note: I have seen one article where the author mentions getting it to work for iOS 6+, but they don't provide any examples: http://zaal.tumblr.com/post/46718877130/why-you-want-to-use-core-data-icloud-sync-if-only-it
As a reference, here are Apple's docs on iCloud + Core Data:
http://developer.apple.com/library/ios/#releasenotes/DataManagement/RN-iCloudCoreData/
http://developer.apple.com/library/ios/#documentation/General/Conceptual/iCloudDesignGuide/Chapters/DesignForCoreDataIniCloud.html
http://developer.apple.com/library/ios/#documentation/cocoa/Conceptual/CoreDataVersioning/vmCloud/vmCloud.html
And here is an example app:
http://developer.apple.com/library/ios/#DOCUMENTATION/General/Conceptual/iCloud101/Introduction/Introduction.html
The Apple developer tutorial on using the iCloud API to manipulate documents might be a good place to start.
Your Third iOS App introduces you to the iCloud document storage APIs. You use these APIs to store and manipulate files in a user’s iCloud storage.
I've been having trouble getting an app submitted to the App Store. This is due to the fact that that database, which is updatable, is too large for the iCloud backup limitations. Most of the data in the db is static, but one table records the user's schedule for reviewing words (this is a vocabulary quiz).
As far as I can tell, I have two or three realistic options. The first is to put the whole database into the Library/Cache directory. This should be accepted, because it's not backed up to iCloud. However, there's no guarantee that it will be maintained during app updates, per this entry in "Make App Backups More Efficient" at this url:
http://developer.apple.com/library/IOs/#documentation/iPhone/Conceptual/iPhoneOSProgrammingGuide/PerformanceTuning/PerformanceTuning.html
Files Saved During App Updates
When a user downloads an app update, iTunes installs the update in a new app directory. It then moves the user’s data files from the old installation over to the new app directory before deleting the old installation. Files in the following directories are guaranteed to be preserved during the update process:
<Application_Home>/Documents
<Application_Home>/Library
Although files in other user directories may also be moved over, you should not rely on them being present after an update.
The second option is to put the data into the NSDocuments or NSLibrary directory, as mark it with the skipBackupFlag. However, one problem is this flag doesn't work for iOS 5.0 and previous per this entry in "How do I prevent files from being backed up to iCloud and iTunes?" at
https://developer.apple.com/library/ios/#qa/qa1719/_index.html
Important The new "do not back up" attribute will only be used by iOS 5.0.1 or later. On iOS 5.0 and earlier, applications will need to store their data in <Application_Home>/Library/Caches to avoid having it backed up. Since this attribute is ignored on older systems, you will need to insure your app complies with the iOS Data Storage Guidelines on all versions of iOS that your application supports
This means that even if I use the "skipBackupFlag", I'll still have the problem that the database is getting backed up to the cloud, I think.
So, the third option, which is pretty much of an ugly hack, is to split the database into two. Put the updatable part into the NSLibrary or NSDocuments directory, and leave the rest in application resources. This would have the small, updatable part stored on the cloud, and leave the rest in the app resources directory. The problem is that this splits the db for no good reason, and introduces possible performance issues with having two databases open at once.
So, my question is, is my interpretation of the rules correct? Am I going to have to go with option 3?
p.s. I noticed in my last post cited urls were edited to links without the url showing. How do I do this?
Have you considered using external file references as described in https://developer.apple.com/library/IOS/#releasenotes/DataManagement/RN-CoreData/_index.html . Specifically, refer to "setAllowsExternalBinaryDataStorage:" https://developer.apple.com/library/IOS/documentation/Cocoa/Reference/CoreDataFramework/Classes/NSAttributeDescription_Class/reference.html#//apple_ref/occ/instm/NSAttributeDescription/setAllowsExternalBinaryDataStorage: . Pushing out large data into a separate file can help reduce database size .
I store some data in my iOS app directly in a local .sqlite file. I chose to do this instead of CoreData because the data will need to be compatible with non-Apple platforms.
Now, I'm trying to come up with the best way to sync this file over iCloud. I know you can't sync it directly, for many reasons. I know CoreData is able to sync its DBs, but even ignoring that using CD would essentially lock this file into Apple platforms (I think? I've only looked into CD a bit), I need the iCloud syncing of this file to work across ALL of iCloud's supported platforms - which is supposed to include Windows. I have to assume that there won't be any compatibility for the CoreData files in the Windows API. Planning out the best way to accomplish this would be a lot easier if Apple would tell us any more than "There will be a Windows API [eventually?]"
In addition, I'll eventually need to implement at least one more sync service to support platforms that iCloud does not. It would be helpful, though not required, if the method I use for iCloud can be mostly reused for future services.
For these reasons, I don't think CoreData can help me with this. Am I correct in thinking this?
Moving on from there, I need to devise an algorithm for this, or find an existing one or an existing 3rd party solution. I haven't stumbled across anything yet. However, I have been mulling over a couple possible methods I could implement:
Method 1:
Do something similar to how CoreData syncs sqlite DBs: send "transaction logs" to iCloud instead and build each local sqlite file off of those.
I'm thinking each device would send a (uniquely named) text file listing all the sql commands that that device executed, with timestamps. The device would store how far along in each list of commands it has executed, and continue from that point each time the file is updated. If it received updates to multiple log files at once, it would execute each command in timestamp order.
Things could get 'interesting' efficiency-wise once these files get large, but it seems like a solvable problem.
Method 2:
Periodically sync a copy of the working database to iCloud. Have a modification timestamp field in every record. When an updated copy of the DB comes through, query all the records with newer timestamps than some reference time and update the record in the local DB from the new data.
I see many potential problems with this method:
-Have to implement something further to recognize record deletion.
-The DB file could get conflicts. It might be possible to deal with them by handling each conflict version in timestamp order.
-Determining the date to check each update from could be tricky, as it depends on which device the update is coming from.
There are a lot of potential problems with method 2, but method 1 seems doable to me...
Does anyone have any suggestions as to what might be the best course of action? Any better ideas than my "Method 1" (or reasons why it wouldn't work)?
Try those two solutions from Ray Wenderlich:
Exporting/Importing data through mail:
http://www.raywenderlich.com/1980/how-to-import-and-export-app-data-via-email-in-your-ios-app
File Sharing with iTunes:
http://www.raywenderlich.com/1948/how-integrate-itunes-file-sharing-with-your-ios-app
I found it quite complex but helped me a lot.
Both method 1 and method 2 seem doable. Perhaps a combination of the two in fact - use iCloud to send a separate database file that is a subset of data - i.e. just changed items. Or maybe another file format instead of sqlite db - XML/JSON/CSV etc.
Another alternative is to do it outside of iCloud - i.e. a simple custom web service for syncing. So each change gets submitted to a central server via JSON/XML over HTTP, and then other devices pull updates from that.
Obviously it depends how much data and how many devices you want to sync across, and whether you have access to an appropriate server and/or budget to cover running such a server. iCloud will do that for "free" but all it really does is transfer files. A custom solution allows you to define your syncing model as you wish, but you have to develop and manage it and pay for it.
I've considered the possibility of transferring a database file through iCloud but I think that I would run into classic problems of timing - slow start for the user - and corrupted databases if the app is run on multiple devices simultaneously. (iPad/iPhone for example).
Sooo. I've had to use the transaction logs method. It really is difficult to implement, but once in place, seems ok.
I am using Apple's SharedCoreData sample as the base for this work. This link requires an Apple Developer Account.
I did find a much much better solution from Tim Roadley however this only works for IOS and I needed both IOS and MacOS.
rant> iCloud development really has to get easier and more stable! /rant