Is it possible to use some level of compression on a xodus store/environment/entities?
I have seen it's possible to encrypt a store. But I haven't seen anything related to compressing a store.
Store encryption is added due to security reasons. Xodus itself tries to store data in an optimal way. Some large portion of data like files/blobs can be stored compressed (zipped) programmatically. You can use VirtualFileSystem for storing large portion of data in a transactional safe way. There is VirtualFileSystem#setCustomConverter for using your own implementation for modifying input/output steams. Check how it used in tests for Snappy compression
Related
Im using a strategy where I'm saving images and pdfs as NSData in the respective managed objects where they belong. I'm having a problem syncing with Ensembles that the pdf doesn't always carry over from one device to another. Now I'm not sure if this is due to some flaws in my code or if it's not a good way of syncing chunks of data like this. Does anyone have experience of this?
I'm using Ensembles 2.2 syncing through CloudKit.
Ensembles should handle this fine. I use it for exactly this purpose, syncing image data including PDF.
I would look closer at the handling of the data. Is the value transformer working (if you are using one)? Is the device capable of unpacking and displaying the PDF data?
An alternative to syncing the PDF directly is transforming to a format like PNG before putting it in your store.
Transformable data type is really just binary under the covers with some additional metadata. Have you tested a simple lightweight migration on an existing store? I suspect the migration would work and would leave the existing data in the store.
If you are looking to get the existing binary data actually moved out of the SQLite file then you are looking at something a bit more involved.
A heavy migration will accomplish what you are looking for but if the stores are large it may take took long and potentially not provide enough feedback for a good user experience. I personally do not use heavy migrations, ever, on IOS but it will accomplish your goal.
An export/import will also work. I generally recommend export/import when a lightweight migration won't work. It involves a medium amount of code but in the end you own the code, understand the entire process and can tweak it to your exact needs.
I have a simple problem to solve but want to know which is better pattern to use and understand the reason for the choice.
Problem :
I want to create a utility which developers can use to check whether the feature should be enabled/disabled depending on the server package version.
eg : server package versions like 10.234, 11.1 etc and client versions 9.3,10.2 etc
Validation on client to see min version on server package for feature to be enabled/disabled.
example data would be like "search lookup feature >= 10.234". And sometimes complex situation need to even check client version whether the feature should be supported.
edit:
Note: Application is very huge and memory is full for most of the time. (thousands of records of organisation data.) So memory is bottle neck.
Just it even flashed to mind to used macro as to do all comparisons and returning value.
I think Plist would be heavy as all the objects would be in NSDictionary and even to access one object, I'll be holding all data.
I want to reduce memory overhead and comparisons too.
For Light data, Use NSUserDefaults or PList. SQLite and Core Data is overkilled.
Updated Answer for Updated Question:-
For server package versions/your app version or other light weight data, you can always use NSUserDefaults or PList.
For records of organisation data, you might want to consider Core Data.
Yes, you can use 2 different types of persistent storage inside your app.
If your data is never going to change, or grow you better use NSUserDefaults, plist or csv file. If you think that user should have the ability to change or add new entries to this data file you better go ahead with CoreData
Plists, csv, and flat files have low overhead for small to medimum amounts of data (and for data where you use all or nearly all) of it.
Systems that only use some of the data on any given run AND have large amounts of data AND have the data structured so SELECT can use an index can be faster (lots and lots faster) with a database (SQLite, or CoreData with the SQL Persistant store).
As a gut feel I expect your problem could be handled well by SQL, but only if you are talking about 1000s of configurations not 10s or even 100s...but you would be better served by benchmarks then my guesses. Go code :-)
I am encrypting downloaded files and saving them locally in app's documents directory.
To read them you must decrypt those file and store some where temporarily.
My concerns are:
1.if I store them in doc directory for time they are being used, for that time window one can get those files using tools like iExplorer.
2.My idea is to store them in memory for the time they are being used and flush the vault after use.
This option is good for small files but for large files say 50 MB or video of 100 MB, I am afraid that app will receive memory warning in result will terminate abruptly.
I want to know the best approach for doing this.
There is no perfect security storing local files in a safe way. If a person has full access to the device, he can always find a way to decrypt the files, as long as your application is able to decrypt it.
The only question is: How much effort is necessary to decrypt the files?
If your only concern is that a person may use iExplorer to copy and open these files, a simple local symmetric encryption will do the trick.
Just embed a random symmetric key in your application and encrypt the data block by block while you download it.
You can use the comfortable "Security Transforms" framework to do the symmetric encryption. There are some good examples in the Apple Documentation.
When you load the files, you can use the same key to decrypt them while you load them from the file system.
Just to make things clear: This is not a perfect protection of the files. But to decrypt the files, one has access to your app binary. Analyse this binary in a debugger and searching for the decryption part to extract your symmetric key. This is a lot effort necessary just to decrypt the files.
Split your files into smaller sizes before saving them, then decrypt on load.
Later edit: I noticed this is mentioned in the comments. I agree splitting files isn't the easiest thing in the world, but presumably you'll only need this for video. About 100MB is a lot of text or audio. If your PDF weights as much, it's probably scanned text, and you can change it into a series if images.
And yes, splitting better be done server-side, don't want the user waste battery in video processing.
Decrypt them, obfuscate them with a toy algorithm (e. g. XOR with a constant block), and store them in documents. When needed, load and decrypt.
Since the problem has no solution in theory (a determined enough attacker can read your process memory after all), it's as good a solution as any.
I need a fast and strong compression + encryption method for my DAT file.
I've a DAT file which contains very sensitive information and I would like to compress and encrypt it. I know I can use Zlib in compression method but how about the encryption method too ?
many thanks
Please check out the Delphi Encryption Compendium (aka DEC):
Another much used component is DCPCrypt:
http://www.cityinthesky.co.uk/opensource/dcpcrypt
It is stream based, so you can layer compression and encryption. I don't know code that does it in one step. (at least not with sensible encryption)
If you need both encryption and compression at the same time, you have two ways of implementing it:
Use your data in a memory buffer, then compress this buffer, then encrypt it;
Use streams, one for compression, the other for encryption.
In all cases, the best is to compress before encryption. It is more difficult to uncypher data from a compressed format, since its content is less expectable.
Then rely on a strong enough encryption algorithm (like AES).
You have all those features in our Open Source units (from Delphi 5 up to XE2). You can use ZIP, or try our much faster (but less efficient in term of compression ration) SynLZ. Then SynCrypto can be used to encrypt it. There are direct functions handling RawByteString kind of data, which contents the data in a memory buffer.
Yes, I know this question is kind of a mouthful of everything...
I have a PDF file on disk, which is encrypted (AES-256, using CommonCrypto/OpenSSL).
I'd like to render the PDF using Quartz' CGPDF... functions and found it's possible to create a CGPDFDocument with a CGDataProvider.
I'm wondering whether it's possible to create callbacks for the CGDataProvider, to decrypt and load only the blocks in memory that Quartz' PDF renderer needs. I'd rather not write the decrypted file to disk from a security/privacy perspective.
Any ideas?
Hopefully I can start an "enlighted" trial & error with the input of stackoverflow's community :)
If it's small enough, you can just keep it in an NSData - decrypt it on the fly and store it in memory. If it's larger, things are more difficult. Ultimately, people also can decrypt your binary and extract the AES key out of it.
PDF's password protection may add a small layer of security, but it's also very easy to circumvent.
What we did eventually is use the CGDataProviderCreateDirect API and implemented the C callbacks to decrypt parts of the encrypted PDF data, which was loaded using [NSData dataWithContentsOfMappedFile:] (which uses mmap() under the hood). This allowed us to render huge encrypted PDFs and having short loading times and low memory usage.