I searched across many threads and issues but I doesn't found the answer.
When I use the Realm Objects Server and the clients connects to the server, the whole DB is synced across all the clients?
In other words, if I have a public DB with millions of objects, relations, etc all the clients have a copy of the whole DB on their devices?
I'll need Realm sync feature but I don't know how to sync occurs. Is the sync incremental? Every user has the objects that device needs to make queries?
My app will increment the size every hour, and the sync feature of the Realm is perfect for me, but I have doubts about the size of DB over the time and how clients sync a lot of data.
Thanks in advance!
At present, yes. Public Realms are synchronized in their entirety to every client's device. The entire list will initially be downloaded when the client's device connects for the first time, and from that point on, any additional changes will be synchronized at the time they are made on the server.
That being said, all Realm files are compacted (ie, all empty allocated space is removed and all strings are condensed), and then compressed with gzip before being downloaded to the client, so as long as the public Realm doesn't contain large binary blobs, even very large files should come down quite quickly.
It's on the roadmap to add partial replication to the Realm Mobile Platform. This will allow only portions of a single Realm file to be synchronized to specific clients. However there are no concrete plans for when this will be released.
For the time being, the easiest thing to do would be to maintain the master list of data on the server, and copy only the desired data to each user's private Realm. However since this would require custom logic to be executed on the server, it would require the Professional or Enterprise editions of the Realm Mobile Platform.
Related
Background story
I am developing a big iOS app. This app works under specific assumptions. The main of them is that app should work offline with internal storage which is a snapshot of last synchronized state of data saved on server. I decided to use CoreData to handle this storage. Every time app launches I check if WiFi connection is enabled and then try to synchronize storage with server. The synchronization can take about 3 minutes because of size of data.
The synchronization process consists of several stages and in each of them I:
fetch some data from the server (XML)
deserialize it
save it in Core Data
Problem
Synchronization process can be interrupted for several reasons (internet connection, server down, user leaving application, etc). This may cause data to be out-of-sync.
Let's assume that synchronization process has 5 stages and it breaks after third. It results in 3/5 of data being updated in internal storage and the rest being out of sync. I can't allow it because data are strongly connected to each other (business logic).
Goal
I don't know if it is possible but I'm thinking about implementing one solution. On start of synchronization process I would like to create snapshot (some kind of copy) of current state of Core Date and during synchronization process work on it. When synchronization process completes with success then this snapshot could overwrite current CoreData state. When synchronization interrupts then snapshot can be simply aborted. My internal storage will be secured.
Questions
How to create CoreData snapshot?
How to work with CoreData snapshot?
How to overwrite CoreDate state with snapshot?
Thanks in advice for any help. Code examples, if it is possible, will be appreciated.
EDIT 1
The size of data is too big to handle it with multiple CoreData's contexts. During synchronization I am saving current context multiple times to cleanup memory. If I do not do it, the application will crash with memory error.
I think it should be resolved with multiple NSPersistentStoreCoordinators using for example this method: link. Unfortunately, I don't know how to implement this.
You should do exactly what you said. Just create class (lets call it SyncBuffer) with methods "load", "sync" and "save".
The "load" method should read all entities from CoreData and store it in class variables.
The "sync" method should make all the synchronisation using class variables.
Finally the "save" method should save all values from class variables to CoreData - here you can even remove all data from CoreData and save brand new values from SyncBuffer.
A CoreData stack is composed at its core by three components: A context (NSManagedObjectContext) a model (NSManagedObjectModel) and the store coordinator (NSPersistentStoreCoordinator/NSPersistentStore).
What you want is to have two different contexts, that shares the same model but use two different stores. The store itself will be of the same type (i.e. an SQLite db) but use a different source file.
At this page you can see some documentation about the stack:
https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/CoreData/InitializingtheCoreDataStack.html#//apple_ref/doc/uid/TP40001075-CH4-SW1
The NSPersistentContainer is a convenience class to initialise the CoreData stack.
Take the example of the initialisation of a NSPersistentContainer from the link: you can have the exact same code to initialise it, twice, but with the only difference that the two NSPersistentContainer use a different .sqlite file: i.e. you can have two properties in your app delegate called managedObjectContextForUI and managedObjectContextForSyncing that loads different .sqlite files. Then in your program you can use the context from one store to get current data to show to the user and you can use the context that use the other store with a different .sqlite if you are doing sync operations. When the sync operations are finally done you can eventually swap the two files and after clearing and reloading the NSPersistentContainer (this might be tricky, because you will want to invalidate and reload all managed objects: you are switching to an entirely new context) you can then show the newly synced data to the user and start syncing again on a new .sqlite file.
The way I understand the problem is that you wish to be able download a large "object graph". It is however so large that it cannot be loaded at once in memory, so you would have to break it in chunks and then merge it locally into to Core data.
If that is the case, I think that's not trivial. I am not sure I can think of direct solution without understanding the object relations and even then it may be really overwhelming.
An overly simplistic solution may be to generate the sqlite file on the backend then download it in chunks; it seems ugly, but it serves to separate the business logic from the sync, i.e. the sqlite file becomes the transport layer. So, I think the essence to the solution would be to find a way to physically represent the data you are syncing in a format that allows for splitting it in chunks and that can afterwards be merged into a sqlite file (if you insist on using Core data).
Please also note that as far as I know Amazon (https://aws.amazon.com/appsync/) and Realm (https://realm.io/blog/introducing-realm-mobile-platform/) provide background sync of you local database, but those are paid services and you would have to be careful not be locked in (should not depend on their libs in your model layer, instead have a translation layer).
We have a mobile application that currently uses a Realm database in conjunction with photos that are stored in the device’s document folder.
We are looking for a way to allow our users to use the app and share data between different devices are were very interested in the new Realm Mobile Platform. One thing we can’t quite figure out though, is how it would work with the photos. Would we need to manually synch those between devices through another service? Or is there a way to make it work with Realm Mobile Platform? Any ideas of how we could make this work would be greatly appreciated.
Storing the photos in the database would work, but might increase the size by a lot. I imagine you also want to preserve offline capabilities - e.g. if a user adds an image while offline, they'll expect that image to eventually sync.
One solution would be to have a per-device realm (i.e. give it a unique name to avoid syncing with anything but the server) that could be used as an image upload queue with a single class (pseudocode):
class Image : RealmObject {
byte[] Data
// Other properties to map this image to whatever
// object it belongs to.
}
Then, your objects with images will look like
class ObjectWithImage : RealmObject {
string ImageUrl
// Other properties
}
So now, when you add a new ObjectWithImage, you'll add a corresponding Image object to the image Realm. Then, on the server, you can listen for changes on the image Realm (requires at least Professional edition) and when a new Image gets synchronized, you can upload the data to AWS S3/Azure blob/other hosting, update the corresponding ObjectWithImage's ImageUrl, and delete the processed Image object. Then on the device you can use an image caching library, e.g. SDWebImage to download the image from the url and store it locally for faster retrieval next time.
The benefits of this solution are:
The shared realm will only contain url's, reducing both storage and bandwidth requirements thus making synchronization faster.
The app will work while offline, eventually syncing both the shared realm and the image realm when connection is restored.
The image Realm will not be synced to other devices, so there will not be any unnecessary traffic.
And, as any solution goes, there are some drawbacks:
It will not work with the Developer edition as it requires modifying the objects on the server.
It may require some more advanced transaction error handling, as you want to commit two transactions in two different realms in a (preferably) atomic fashion.
Newbie question.
I will need to have a data base from about 200 UIImages (single of them less than 500kb size) for iPad app. Customer want to have possibility to change set of this images from time to time without releasing new version of app in appstore and app must work without connection to the web (local data base on a device). I don't see how this can be done simultaneously, I see only one common option here:
Image data base would be stored on a server, what app customer will be able to change anytime. User will need to have web connection and every time he will start the application - existing data base will load into the app.
Main questions here:
is it possible to update data base on user's device without releasing new version of app and what data base managing system is more proper to this situation(SQLite, MySQL etc...)?
Q : is it possible to update data base on user's device without releasing new version of app?
A : Yes. It is possible.
SQLite will be perfect for you.
The photographs reside on the web server.
A number of start-off photographs may reside within the boundle so that the app is not really empty at start.
However, when downloading the app, the user must be online. In most cases he would still be online directly afterwards when he launches the app for the first time.
The server provides two services:
A quick one that just provides a version number of the
photo-database content and/or the date of the last change to the
photographs on the server.
The app frequently (not more than daily I would say) checks wether there are new images on the server or not.
If they are then the user is asked, whether he wants to download them.
If the user says YES then the app sends the version number and/or last date and/or IDs of all local photographs to the server and the
server provides the information about which photographs have been
added and where to download that very photograph and which have to
be deleted.
Then you add or delete or update the photographs from the download source given by the server. (That may well be an URL to the
very same server of course.)
For 200 data sets I would strongly suggest core data with SQLite - the standard stuff.
You may then think of holding the image data in the file system or in NSData properties within the database.
I need to define a large amount of data to be stored within an app and used as a lookup table. For instance, I have an array of manufacturer names, each with a mfg code. Each manufacturer can make different products, each with their own code as well.
A,7 could be deciphered to mean
Manufacturer: Apple(A)
Product: MacMini(7)
I see several ways of defining this, but I'm not sure which would be best.
Option 1) #define these constants in a separate header file such as:
#define MFG_APPLE #"A"
#define MFG_DELL #"B"
#define PRODUCT_MAC_MINI 7
#define PRODUCT_INSPIRON 2
Option 2) create a dictionary object filled with dictionary objects to allow me to index through them easier.
Option 3) use core data to create a database of these mfgs and products and relationships.
If option 2 or 3 is suggested, are there easy ways to pre-populate these data structures instead of hard-coding them to populate during program startup?
Option 4) Create a web service to tie this back to a server, where the data can be updated more often. A JSON query will send the mfg and product codes to the server, where it can respond with the mfg and product names.
You should consider the following: If the database is shipped with the app, you will have to release an update for the app each time the database must be updated. So the question is, how frequently will you have to update the data? If it's fine to update the database once every couple of months or maybe just once a year, shipping the database with your app might be an option, if you need to update it every month or even weekly, you should definitely host the database somewhere on the web; releasing an update in such short intervals is not a feasible option.
Another thing you should consider: If the database exists solely as a web service and each look up requires a JSON call to the server, it won't be possible to perform a lookup if the user is offline (currently has no network access for whatever reason). Also each lookup costs the user traffic, so if the user has a monthly limit, yet needs to perform plenty of lookups a day, using your app may quickly cause him to exceed that monthly limit, leaving him without any Internet service (or a very throttled one) in the end.
From my experience, it is best to host such a database online, yet cache it for offline access if possible. The app itself ships with a database copy, that was up-to-date the day you built the app for distribution. Each time the app is started, and maybe once a day in case the app is never quit, it will query a web server for the current "version" of the database. If this version is newer than the one shipped with the app, it tries to downloads a copy of this database to its local cache and then switches to the cached copy for future lookups. If the cached copy gets lost (caches may be flushed by the system at any time), it will have to re-download it. In the meantime, it can use the shipped database, which is outdated, yet better than nothing. If download is not possible (e.g. not enough free space is available on the device), the app may want to make online queries directly if the user currently is online, fall back to the out-dated shipped database if he is offline, and retry to download a cache copy at some later time (maybe the device will have more free space available at that time).
So basically your app will have a work flow as follows:
START
A locally cached copy exists? If NO Goto 6.
The locally cached copy is up-to-date? If NO Goto 5.
Perform the lookup using the local cached copy. Goto 12.
Delete the outdated cached copy. Goto 1.
The shipped database is up-to-date? If NO Goto 8.
Perform the lookup using the shipped database. Goto 12.
Download the updated database.
Download succeeded? If YES Goto 4.
The user is currently online? If NO Goto 7.
Perform the lookup using a JSON webservice. Goto 12.
END
If you only add more entries to the database in the future, yet existing entries will never change, there is also another, even much better option: You have simply two databases. One that ships with the app and one that only contains the updates (new entries added) after the last app release. This shrinks the amount of data that needs to be downloaded and cached dramatically. In that case your app must always perform two lookups, one in the shipped database (which is always performed first), and if nothing is found there, in the downloaded cached copy, which does not contain the entries already found in the shipped database (or directly online, if no cached copy is available, yet the user currently has Internet access). Each time you release a new update of the app, it will get a new full copy of the database, hence you can reset the update database back to zero entries and only keep adding new entries there (or you can keep different update databases lying around on the server for different app versions that had different databases shipped with them, if you don't think that is too much hassle to manage).
The update database for download may even be created dynamically by the server, that would of course be the best option. E.g. after shipping the app, you add 3 vendors and 30 products to the database, and every vendor and product has a unique ID (that is strictly increasing with each new entry added), then the app can tell the server that the highest vendor it knows has ID X and the highest product has ID Y, in which case the server sends out an update database with all vendors and products whose IDs are higher than X and Y.
All these decisions influence on the database format to use. Generally it sounds a lot like a job for CoreData, yet if you want dynamic update databases, the updates should be delivered in a different format (JSON, XML, CVS, or something else a server can easily generate) and be converted to CoreData by the app after the download is completed, since dynamically generating CoreData databases on a server is rather hard and definitely not recommend.
I've already managed to program a webapp for personal use I'm really satisfied with. Not being something meant for public usage and distribution, I didn't want to go through the hassle of jailbreaking my device just to be able to run my own application, so I made this seamlessy looking and behaving webapp (and of course I've added it to the other apps saving it as a "Home application")
Since the start time can be a bit slow and I'm constantly pushing my data from and to a remote server, can I force the usage of html5 offline browsing (with a cache manifest) even when I am online? Also, I'm thinking of persisting the data as local storage and from time to time synch it to the server. Since I've never used html5 local storage, how much reliable is it? Can I lose my data?
Is this a viable pattern to quickly create a personal iPhone app? Thanks
Yes, you can force the usage.
so basically you should a very simple checking :
if(localStorage["mycontent"]!==null)
{
// do it offline.
}else
{
// retrieve from server database
}
For your question regarding :
Also, I'm thinking of persisting the data as local storage and from
time to time synch it to the server. Since I've never used html5 local
storage, how much reliable is it? Can I lose my data?
The answer is it depends. If the data is static (or can only be changed by you and not other user ) it's reliable. You also have to take note when a data can be considered expired so localstorage can be filled with refreshed data from the server.
But take note that cleaning history is also remove your data, so only use Localstorage as a cache/mirror of the data in the server.
window.localStorage.setItem('x',y);
window.localStorage.getItem('x';
window.localStorage.removeItem('x');
Lets you store, read and delete persistent data in HTML5. See https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API/Using_the_Web_Storage_API
But note that on IOS Safari puts this data in the cache folder which on occasions gets expunged. So do plan a server sync and restore of this data if important.
Alternatively use a local SQLite database for a more persistent persistence....