Recently we added a change in our app that changed the resource addition to Photos - Until now we called addResourceWithType:fileURL:options: with PHAssetResourceCreationOptions.shouldMoveFile set to YES, and when we changed it to NO (the default value) we observed much more asset creation failures. Specifically we see a new error code in the procedure - PHPhotosErrorNotEnoughSpace. One can clearly see a connection between adding more storage to the file system and an asset creation failure that is related to storage, but we are struggling to understand a few things:
The storage of the device is always higher than the video size, usually by a great amount - We observed failures for devices with 120GB free storage, while the video size was 200MB.
Generally we save quite a lot of resources to the file system, so it is quite surprising to see supposedly storage issues when adding a relatively low amount of extra storage.
The asset creation is part of a bigger procedure of encoding a video to a file system, and then moving/copying it to Photos. Is it that big of a difference to copy a video of 100MB-200MB instead of moving it, such that the overall procedure failure will increase drastically?
Appreciate any help.
Related
A piece of software I'm working on outputs quite a lot of files which are the stored on a server. During its runtime I've had one file go corrupt on me. These files are critical to the operation, so this cannot happen. I'm therefore trying to come up with a way of adding error correction to the files to prevent this from ever happening again.
I've read up on Reed-Solomon, which encodes k blocks of data plus m blocks of parity, and can then reconstruct up to m missing blocks. So what I'm thinking is taking the data stream, split it into these blocks, and then store them in sequence on disk, first the data blocks, then the parity blocks. Repeat until entire file is stored. k, m, and block sizes are of course variables I'll have to investigate and play with.
However, it's my understanding that Reed-Solomon requires you to know which blocks are corrupt. How could I possibly know that? My thinking is I'd have to add some extra, simpler, error detection code to each of the blocks as I write them, otherwise I can't know if they're corrupted. Like CRC32 or something.
Have I understood this correctly, or is there a better way to accomplish this?
This is a bit of an older question, but (in my mind) is always something that is useful and in some cases necessary. Bit rot will never be completely cured (hush ZFS community; ZFS only has control of what's on it's filesystem while it's there), so we always have to come up with proactive prevention and recovery plans.
While it was designed to facilitate piracy (specifically storing and extracting multi-GB files in chunks on newsgroups where any chunk could go missing or be corrupted), "Parchives" are actually exactly what you're looking for (see the white paper, though don't implement that scheme directly as it has a bug and newer schemes are available), and they work in practice as follows:
The complete file is input in to the encoder
Blocks are processed and Reed-Solomon blocks are generated
.par files containing those blocks are output along side the original file
When integrity is checked (typically on the other end of a file transfer), the blocks are rechecked and any blocks that need to be used to reconstruct missing data are pulled from the .par files.
Things eventually settled in to "PAR2" (essentially a rewrite with additional features) with the following scheme:
Large file compressed with RAR and split in to chunks (typically around 100MB each as that was a "usually safe" max of usenet)
An "index" file is placed along side the file (for example bigfile.PAR2). This has no recovery chunks.
A series of par files totaling 10% of the original data size are along side in increasingly larger filesizes (bigfile.vol029+25.PAR2, bigfile.vol104+88.PAR2, etc)
The person on the other end can then gets all .rar files
An integrity check is run, and returns a MB count of out how much data needs recovery
.PAR2 files are downloaded in an amount equal to or greater than the need
Recovery is done and integrity verified
RAR is extracted, and the original file is successfully transferred
Now without a filesystem layer this system is still fairly trivial to implement using the Parchive tools, but it has two requirements:
That the files do not change (as any change to the file on-disk will invalidate the parity data (of course you could do this and add complexity with a copy-on-change writing scheme))
That you run both the file generation and integrity check/recovery when appropriate.
Since all the math and methods are both known and battle-tested, you can also roll your own to meet whatever needs to have (as a hook in to file read/write, spanning arbitrary path depths, storing recovery data on a separate drive, etc). For initial tips, refer to the pros: https://www.backblaze.com/blog/reed-solomon/
Edit: The same research that led me to this question led me to a whole subset of already-done work that I was previously unaware of
https://crates.io/crates/solana-reed-solomon-erasure (as well as a bunch of other implementations in the Rust crate registry)
https://github.com/klauspost/reedsolomon (based on the BackBlaze code, and processes 1Gbps per core)
Etc. Look for "Reed-Solomon file recovery "
I used log4j (v. 1) in the past and was glad to know that a major refactoring was done to the project, resulting in log4j 2, which solves the issues that plagued version 1.
I was wondering if I could use log4j 2 to write to data files, not only log files.
The application I will be soon developing will need to be able to receive many events from different sources and write them very fast either to a data file or to a database (I haven't decided which yet).
The thread that receives the events must not be blocked by I/O while attempting to write events, so log4j2's Asynchronous Loggers, based on the LMAX Disruptor library, will definitely fit this scenario.
Moreover, my application must be able to recover either from a 'not enough space on disk' or 'unable to reach database' conditions, when writing to a data file or to a database table, respectively. In other words, when the application runs out of disk space or the database is temporarily unavailable, my application needs to store events in memory and wait for storage to become available and when it does, write all waiting events to disk or database.
Do you think I can do this with log4j?
Many thanks for your help.
Regards,
Nuno Guerreiro
Yes.
I'm aware of at least one production implementation in a similar scenario, where in gathered events are written to disk at high throughput.
Write to a volume other than your system volume to minimize the chances of system crashes due to disk space overrun.
Upfront capacity planning can help in ensuring h/w configuration with adequate resources to handle projected average load and bursts, for a reasonable period of time.
Do not let the system run out of disk space :). Keep track of disk usage, and proactively drop older data in extreme circumstances.
My app has to make sure that there is always a minimum amount (maybe 5-10 MB) of free disk space available, for it to operate correctly. I have to save important data to file, otherwise everything is going to be corrupted.
The main problem is: What if the user downloads my app and he had EXACTLY that amount of disk space free that he needed for the app (without my on first startup created data)
When he's now starting the app I try to do some initial setup stuff - here I have to write around 2-3 MB to file and this space must be free.
What can I do to achieve this?
You can do two things:
First, when your application starts, you can find out the available space using NSFileManager.attributesForFilesystemofPath:. If the amount of space is low (and 5 MB free is really low, you can warn the user and maybe even refuse to start actions in your app that require this available space.
Second, you could create a scratch file. If you know you need 5MB storage, then maybe you can allocate a 5MB file and use that file for your storage.
You don't specify what you need to store on disk but if it were for example one file then you could write this data over your 5MB scratch file instead of opening a new file.
This requires some extra bookkeeping, but you do get guaranteed space for your data.
In short, you can't. You will need to write your code to have an initial startup that checks to see if there is enough disk space to do your initial setup, and display an error if not. I haven't tried to do this on iOS before so I'm not sure what the sandbox allows you to do off the top of my head. I'd have to go read up in the iOS file system docs. That's what I suggest you do.
I have got an iPhone application where I archive permanent data in the documents folder of the application (arrays, dictionaries). I read/write from and to the documents folder quite frequently and I would like to know whether this is considered a bad habit. Wouldn't it be better if I had a singleton class, read and write to arrays there and then, only when the application quits, write this data to the documents folder ? I do not see/feel any performance issues right now on my iPhone 5, but I wanted to know whether this is a bad practise.
FLASH memory has limited write capability - a long time ago it was rated in some increment of thousands. Not sure where it is today.
That said, if your app is using the standard file system APIs, then the system is using the file cache, and you might open a file, read it then change it many times without the file system ever writing to flash. The system may sync to flash occasionally, but that process is opaque - no way to really know when or why iOS does it.
The UNIX APIs allow for syncing the file system cache to the storage system (iOS this is FLASH), but if you are not using that then you are probably not doing much I/O at all given what you say above.
Given the lack of Apple discouraging developers from writing to the file system, I for sure would not worry about this.
But, if you said you were writing gigabytes of image data every few minutes - well - that might be a problem.
The iOS app I'm working on handles very large files (largest being 2.7 gig in size), and some sqlite tables, after downloading the large I'm getting console messages like this one :
Aug 24 14:50:28 unknown TheAppName[1407] <Warning>: Error opening database: 14 unable to open database file
Also other issues that happen, loading a view (which has already been loaded and presented) will cause a crash saying "nib cannot be located".
The app can run without the large file just fine, but it is very useful to have. The question I have is this : Is the large file causing some sort of memory issue, or is there some sort of limit that iOS places on files? I've looked around at other places and I've been unable to find anything specific to this.
Does it work in the Simulator or do you also have the same issue there?
I suggest you run your app using Instruments with the ObjectAlloc tool. Its likely you have some leaks in the way you are managing the download. Or, is there free space on the device for this file?
Others have had success with downloading huge files as long as sufficient space exists on the flash. You can test for this - there are a few functions floating around that tell you how to determine free disk space.
Another idea: write your file to the disk with the "F_NOCACHE" (fcntl) flag set on the file descriptor. This will cause the writes to bypass the "uniform buffer queue", which is used to cache disk blocks and for other memory uses, and will greatly reduce stress on the system itself.
An iOS app's max size limit is 2GB, this is limit is causing all the errors and is why your app is crashing. I would recommend you upload your large size files to a file hosting site and stream them into the app when you need them.