The situation
I am using Core Data's "Allows External Storage" to store compressed images and small audio files in Core Data. Performance benchmarks have shown that this is actually quite performant. Also, I am using Core Data's PersistentCloudKitContainer to sync my database with iCloud.
"Allows External storage" will automatically save files that are bigger than ~500KB (?) to the file system and only store a reference in the database. This works nicely. For instance, a 1MB image file is stored as an external record and takes up the expected 1MB of iCloud storage after syncing. Also, files that are smaller than those ~500KB are not stored as external record (ckAsset), but as binaryData in the database record.
The problem:
For some reason a 0.47MB binary data file that is stored directly in the database will take up about 4.3MB of iCloud storage. That is 9x of the expected amount. Inspecting the binary data stored in the record shows that the binary data itself has the expected size of only 0.47MB (CloudKit Dashboard). Also, I have verified that the local app bundle only grows by the expected 0.47MB. Thus, how can those additional 3.8MB of consumed iCloud storage be explained? In contrast, audio and image files that are larger than ~500KB are stored as external records and take up the correct amount of iCloud storage.
Please look at this annotated image for a better understanding:
Image that illustrates the problem (CloudKit Dashboard)
Ideas / Workarounds / What I tried:
I could try to find a way to always store files as ckAssets/external records. (e.g. by lowering the limit for storing ckAssets to 0.01MB). If that is possible.
Could the Write-Ahead-Log (WAL) of SQLite be involved in creating huge temporary sqlite-wal files? I tried limiting the WAL journal size and the local sqlite-wal is small, so I don't think that this is where the problem lies. Unless there is an independent iCloud WAL file that I don't know about.
I would be glad if anyone could help me with this issue. Thanks in advance!
The common method to store images in a database is to convert the image to base64 data before storing the data. This process will increase the size by 33%. Alternatively it is possible to directly store the image as a BLOB; for example:
$image = new Imagick("image.jpg");
$data = $image->getImageBlob();
$data = $mysqli->real_escape_string($data);
$mysqli->query("INSERT INTO images (data) VALUES ('$data')");
and then display the image with
<img src="data:image/jpeg;base64,' . base64_encode($data) . '" />
With the latter method, we save 1/3 storage space. Why is it more common to store images as base64 in MySQL databases?
UPDATE: There are many debates about advantages and disadvantages of storing images in databases, and most people believe it is not a practical approach. Anyway, here I assume we store image in database, and discussing the best method to do so.
I contend that images (files) are NOT usually stored in a database base64 encoded. Instead, they are stored in their raw binary form in a binary column, blob column, or file.
Base64 is only used as a transport mechanism, not for storage. For example, you can embed a base64 encoded image into an XML document or an email message.
Base64 is also stream friendly. You can encode and decode on the fly (without knowing the total size of the data).
While base64 is fine for transport, do not store your images base64 encoded.
Base64 provides no checksum or anything of any value for storage.
Base64 encoding increases the storage requirement by 33% over a raw binary format. It also increases the amount of data that must be read from persistent storage, which is still generally the largest bottleneck in computing. It's generally faster to read less bytes and encode them on the fly. Only if your system is CPU bound instead of IO bound, and you're regularly outputting the image in base64, then consider storing in base64.
Inline images (base64 encoded images embedded in HTML) are a bottleneck themselves--you're sending 33% more data over the wire, and doing it serially (the web browser has to wait on the inline images before it can finish downloading the page HTML).
On MySQL, and perhaps similar databases, for performance reasons, you might wish to store very small images in binary format in BINARY or VARBINARY columns so that they are on the same page as the primary key, as opposed to BLOB columns, which are always stored on a separate page and sometimes force the use of temporary tables.
If you still wish to store images base64 encoded, please, whatever you do, make sure you don't store base64 encoded data in a UTF8 column then index it.
Pro base64: the encoded representation you handle is a pretty safe string. It contains neither control chars nor quotes. The latter point helps against SQL injection attempts. I wouldn't expect any problem to just add the value to a "hand coded" SQL query string.
Pro BLOB: the database manager software knows what type of data it has to expect. It can optimize for that. If you'd store base64 in a TEXT field it might try to build some index or other data structure for it, which would be really nice and useful for "real" text data but pointless and a waste of time and space for image data. And it is the smaller, as in number of bytes, representation.
Just want to give one example why we decided to store image in DB not files or CDN, it is storing images of signatures.
We have tried to do so via CDN, cloud storage, files, and finally decided to store in DB and happy about the decision as it was proven us right in our subsequent events when we moved, upgraded our scripts and migrated the sites serveral times.
For my case, we wanted the signatures to be with the records that belong to the author of documents.
Storing in files format risks missing them or deleted by accident.
We store it as a blob binary format in MySQL, and later as based64 encoded image in a text field. The decision to change to based64 was due to smaller size as result for some reason, and faster loading. Blob was slowing down the page load for some reason.
In our case, this solution to store signature images in DB, (whether as blob or based64), was driven by:
Most signature images are very small.
We don't need to index the signature images stored in DB.
Index is done on the primary key.
We may have to move or switch servers, moving physical images files to different servers, may cause the images not found due to links change.
it is embarrassed to ask the author to re-sign their signatures.
it is more secured saving in the DB as compared to exposing it as files which can be downloaded if security is compromised. Storing in DB allows us better control over its access.
any future migrations, change of web design, hosting, servers, we have zero worries about reconcilating the signature file names against the physical files, it is all in the DB!
AC
I recommend looking at modern databases like NoSQL and also I agree with user1252434's post. For instance I am storing a few < 500kb PNGs as base64 on my Mongo db with binary set to true with no performance hit at all. Mongo can be used to store large files like 10MB videos and that can offer huge time saving advantages in metadata searches for those videos, see storing large objects and files in mongodb.
In my app, I want to keep very sensitive data persisted on a client in an encrypted cache, and thought of using the keychain.
Potentially, we could end up putting quite a bit of information (a couple of MBs) into this cache and was wondering...
Are there any hard limits on the size of data that I can cram into the keychain?
Is there another/better place I can store this data? I only need a simple key/value interface similar to NSUserDefaults, but encrypted.
Thanks in advance!
The keychain (consider the name) is designed to hold keys and other reasonably small secure items. For data, encrypt it with AES using Common Crypto and save the key in the keychain. Create the key from random bytes. Save the encrypted data in the Documents directory or subdirectory.
We need to store various data (accesstokens, receipts). In bytes this is relatively small (20000 symbols or so).
We don't want the user to be able to read and tamper this data because we to some extent don't want any smart users to bypass our systems in some way.
We don't want this data to be stored after app is deleted. Therefore keychain seems inappropriate. This is wanted because it seems sensible that the user should get a clean install every time they install the app.
If you want to have the data secure you should use Core Data with apples Data Protection on the DB file.
In addition to that you should encrypt the data itself too.
UPDATE:
You may want to give this a look for encrypting the data: RNCryptor
And this for Data Protection: Data Protection
I want to create an in memory data store with core data on the iphone in the following way:
The data of the store is saved to disk in an encrypted file (max size 400kb)
The encrypted file is loaded completly into memory and afterwards I will decrypt it so that I have some data array in memory
I want to tell the NSPersistentStoreCoordinator to use this data array which is the store I want to use.
At certain points in the code the current in memory data store will be copied to another data array, encrypted and stored to disk such that the data on disk corresponds always to the most recent version of the data.
I must do that because the data is sensitive user data that absolutly cannot be stored in a plain database.
In my app I already implemented a version where each property of the managed objects are encrypted, such that the sqlite database which is stored on disk contains only cryptic unreadable values. Unfortunatly it turned out to be too slow to encrypt an decrypt everything everytime on the fly.
First off: Is this possible?
Secondly: Might there be some things I need to pay attention to?
I'm not sure if this will be of any help to you but in iOS5 persistent stores now store data in an encrypted format on disk. This is also an option in iOS4. See the documentation.
For applications built for iOS 5.0 or later, persistent stores now store data by default in an encrypted format on disk. The default protection level prevents access to the data until after the user unlocks the device for the first time. You can change the protection level by assigning a custom value to the NSPersistentStoreFileProtectionKey key when configuring your persistent stores. For additional information about the data protection that are new in iOS 5.0, see “Data Protection Improvements.”