Using GCD for notification of end of file write? [duplicate] - ios

I'm working on iOS project that supports iTunes file sharing feature. The goal is realtime tracking incoming/changed data's.
I'm using (kinda modified) DirectoryWatcher class from Apple's sample code
and also tried this source code.
The data is NSBundle (*.bundle) and some bundles are in 100-500 MB ranges, depends on its content, some video/audio stuff. The bundles has xml based descriptor file in it.
The problem is any of these codes above fires notification or whatever else when the data just started copying and but not when the copy/change/remove process finished completely.
Tried next:
checking file attributes:
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
BOOL fileBusy = [[fileAttrs objectForKey:NSFileBusy] boolValue];
looking for the fileSize changes:
dispatch_async(_checkQueue, ^{
for (NSURL *contURL in tempBundleURLs) {
NSInteger lastSize = 0;
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
NSInteger fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
do {
lastSize = fileSize;
[NSThread sleepForTimeInterval:1];
fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
NSLog(#"doing job");
} while (lastSize != fileSize);
NSLog(#"next job");
}
);
any other solutions?
The solution above works great for bin files, but not for .bundle (as .bundle files are directory actually). In order to make it work with .bundle, you should iterate each file inside .bundle

You can use GCD's dispatch sources mechanism - using it you can observe particular system events (in your case, this is vnode type events, since you're working with file system).
To setup observer for particular directory, i used code like this:
- (dispatch_source_t) fileSystemDispatchSourceAtPath:(NSString*) path
{
int fileDescr = open([path fileSystemRepresentation], O_EVTONLY);// observe file system events for particular path - you can pass here Documents directory path
//observer queue is my private dispatch_queue_t object
dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_VNODE, fileDescr, DISPATCH_VNODE_ATTRIB| DISPATCH_VNODE_WRITE|DISPATCH_VNODE_LINK|DISPATCH_VNODE_EXTEND, observerQueue);// create dispatch_source object to observe vnode events
dispatch_source_set_registration_handler(source, ^{
NSLog(#"registered for observation");
//event handler is called each time file system event of selected type (DISPATCH_VNODE_*) has occurred
dispatch_source_set_event_handler(source, ^{
dispatch_source_vnode_flags_t flags = dispatch_source_get_data(source);//obtain flags
NSLog(#"%lu",flags);
if(flags & DISPATCH_VNODE_WRITE)//flag is set to DISPATCH_VNODE_WRITE every time data is appended to file
{
NSLog(#"DISPATCH_VNODE_WRITE");
NSDictionary* dict = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:nil];
float size = [[dict valueForKey:NSFileSize] floatValue];
NSLog(#"%f",size);
}
if(flags & DISPATCH_VNODE_ATTRIB)//this flag is passed when file is completely written.
{
NSLog(#"DISPATCH_VNODE_ATTRIB");
dispatch_source_cancel(source);
}
if(flags & DISPATCH_VNODE_LINK)
{
NSLog(#"DISPATCH_VNODE_LINK");
}
if(flags & DISPATCH_VNODE_EXTEND)
{
NSLog(#"DISPATCH_VNODE_EXTEND");
}
NSLog(#"file = %#",path);
NSLog(#"\n\n");
});
dispatch_source_set_cancel_handler(source, ^{
close(fileDescr);
});
});
//we have to resume dispatch_objects
dispatch_resume(source);
return source;
}

I found two rather reliable (i.e. not 100% reliable but reliable enough for my needs) approaches, which only work in conjunction with polling the contents of the directory:
Check NSURLContentModificationDateKey. While the file is being transferred, this value is set to the current date. After transfer has finished, it is set to the value the original file had: BOOL busy = (-1.0 * [modDate timeintervalSinceNow]) < pollInterval;
Check NSURLThumbnailDictionaryKey. While the file is being transferred, this value is nil, afterwards it cointains a thumbnail, but probably only for file types from which the system can produce a thumbnail. Not a problem for me cause I only care about images and videos, but maybe for you. While this is more reliable than solution 1, it hammers the CPU quite a bit and may even cause your app to get killed if you have a lot of files in the import directory.
Dispatch sources and polling can be combined, i.e. when a dispatch source detects a change, start polling until no busy files are left.

Related

Write logs in HTML file using GCD doesn't work as expected

I created an overload method for NSLog in order to write all logs in an HTML file.
All logs are written in the HTML file using GCD.
The problem is that lines are sometimes truncated...
Here my code :
Write in log file function :
+(void)writeInLogFile:(NSString *)strLog inFolder:(NSString *)folder fileName:(NSString *)fileName extension:(NSString *)extension{
//Create Directory
NSString *path;
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
path = [[paths objectAtIndex:0] stringByAppendingPathComponent:folder];
NSError *error;
if (![[NSFileManager defaultManager] fileExistsAtPath:path]) //Does directory already exists?
{
if (![[NSFileManager defaultManager] createDirectoryAtPath:path withIntermediateDirectories:YES attributes:nil error:&error])
{
NSLog(#"%d||Create log directory error: %#",LOG_SEVERITY_HIGH, error);
}
}
NSString* filePath = [NSString stringWithFormat:#"%#/%#.%#",path,fileName,extension];
int nbOfLogFiles = [[[NSFileManager defaultManager] contentsOfDirectoryAtPath:path error:nil] count];
if(nbOfLogFiles <= NB_OF_LOG_FILES_BEFORE_PURGE +1){
[HandleString createLogFile:filePath andStrLog:strLog];
}
else{
[HandleString purgeLogDirectory:path];
[HandleString createLogFile:filePath andStrLog:strLog];
}
}
That is the result :
And the call (called each time an NSLog is executed):
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0), ^{
[HandleString writeInLogFile:message];
});
That is the result :
As you can see, the last line is truncated...
This appens throughout the file.
I tried to run the process on the main thread and it works well without problems.
Another interesting thing when i change the QOS the result isn't the same, for exemple whith priority high, i have more truncated lines.
Edit : The code to write in file :
+(void)createLogFile:(NSString*)filePath andStrLog:(NSString*)strLog{
if(![[NSFileManager defaultManager] fileExistsAtPath:filePath]){
[[NSFileManager defaultManager] createFileAtPath:filePath contents:nil attributes:nil];
}
NSFileHandle *fileHandle = [NSFileHandle fileHandleForWritingAtPath:filePath];
[fileHandle seekToEndOfFile];
[fileHandle writeData:[strLog dataUsingEncoding:NSUTF8StringEncoding]];
[fileHandle closeFile];
}
The issue is that you're writing from multiple threads simultaneously. Consider this sequence of events:
Thread A seeks to end of file
Thread B seeks to end of file
Thread B writes its data
Thread A writes its data
Thread A's data will overwrite Thread B's data. If Thread A's data is longer, then no trace of Thread B's data will be left. If Thread A's data is shorter, then the part of Thread B's data beyond that length will remain.
Swapping the order of the last two steps has a similar problem. And there are, of course, more complicated scenarios with more threads.
One solution is to serialize all accesses to the file, as you've done in your self-answer. Another is to open the file in append-only mode. In this mode, all writes are done at the end of the file. This is enforced by the OS. There's no way for two simultaneous writes to overwrite each other. The current file position of the file handle is irrelevant, so there's no need to seek.
NSFileHandle doesn't directly support opening a file in append-only mode. You have to use open() to open a file descriptor and then hand that file descriptor off to NSFileHandle. (You could also just use write() and close(), but it's a bit messier.) For example:
int fd = open(filePath.fileSystemRepresentation, O_WRONLY | O_APPEND | O_CREAT, S_IRUSR | S_IWUSR);
if (fd < 0)
{
// handle error
}
NSFileHandle* fileHandle = [[NSFileHandle alloc] initWithFileDescriptor:fd closeOnDealloc:YES];
[fileHandle writeData:[strLog dataUsingEncoding:NSUTF8StringEncoding]];
[fileHandle closeFile];
[fileHandle release]; // Only needed if not using ARC
Note that you don't have to explicitly create the log file in this case. The open() call will create it if it doesn't exist, because of the O_CREAT flag.
Given the phrase "I tried to run the process on the main thread and it works well without problems." I would say that your truncation problem comes from a buffer being closed before completely empty. That does not explain the disparity you might have observed while using high priority (remember that the extra lines might be coincidental until proven otherwise...).
Now, I would recommend you to try a small sleep before ending the thread scope...
I found a solution a lil bit tricky :
I removed the dispatch_asynch :
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0), ^{
[HandleString writeInLogFile:message];
});
And in my write in log file function i did :
static dispatch_queue_t asyncQueue;
static dispatch_once_t onceToken;
dispatch_once(&onceToken, ^{
asyncQueue = dispatch_queue_create("asyncQueue", NULL);
});
if(nbOfLogFiles <= NB_OF_LOG_FILES_BEFORE_PURGE +1){
dispatch_async(asyncQueue, ^{
[HandleString createLogFile:filePath andStrLog:strLog];
});
}
else{
dispatch_async(asyncQueue, ^{
[HandleString purgeLogDirectory:path];
[HandleString createLogFile:filePath andStrLog:strLog];
});
}
It could may be help someone.
But i have to know if according to you, it is a good solution?
John

iOS Background Transfer - com.apple.nsurlsessiond folder full of tmp files

We've written a media application that allows you to get a list of latest videos as json list using BACKGROUND FETCH
then it uses BACKGROUND TRANSFER to tell iOS to download the video one by one and go back to sleep and to wake the app when its done.
It does all that but we've noticed that Space Usage is growing and growing.
We added code to clear all downloaded videos but space usage stayed hi in settings.
We downloaded the app folders using Xcode > Organizer> Devices and found the BACKGROUND TRANSFER tmp folder was dull of tmp files.
Shouldn't these be getting cleared out
This is in general the code I use.
I think the main is I attach multiple DownloadTask(can be up to 30) to one background session. files vary in size from movies to pdfs.
NSURLSession * backgroundSession_ = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier:identifier];
backgroundSession_ = [NSURLSession sessionWithConfiguration:urlSessionConfigurationBACKGROUND_
delegate:self
delegateQueue:[NSOperationQueue mainQueue]];
NSOperationQueue *mainQueue_ = [NSOperationQueue mainQueue];
NSURLSessionDownloadTask * downloadTask_ = [backgroundSession_ downloadTaskWithURL:url_];
downloadStarted_ = TRUE;
[downloadTask_ resume];
Try something like this before returning from didFinishDownloadingToURL:
// App's Documents directory path
NSString *docsPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)firstObject];
// Creating the path for the downloaded file
docsPath = [docsPath stringByAppendingPathComponent:downloadTask.response.suggestedFilename];
// Moving the file from temp location to App's Documents directory
[[NSFileManager defaultManager] moveItemAtPath:location.path toPath:docsPath error:NULL];
The documentation states that you should "move the file to a permanent location in your app’s sandbox container directory before returning from this delegate method" (perhaps the Documents directory).
The temp files that gets cleared out after you return from didFinishDownloadingToURL (or if the download failed) - at the OS's discretion (usually under memory pressure).
I have the same issue but in a bit different circumstances:
on older devices (iPhone 4S or older) the app is usually killed during the background fetching by the OS. Probably to free memory. In this case the tmp files are retained (and untracked). Next time the app has an opportunity to fetch, new files are created... and this cycle goes on and on till the user recognises the app uses 4gb of storage space - and deletes it.
I haven't found the perfect solution yet - even after I set the background configuration's -NSURLSessionConfiguration URLCache to a custom one (documentation says it's nil by default) with a path the same directory (defaultCacheDir/com.apple.nsurlsessiond/...) was used - but made a cleanup method and use it when I sure there is no download in progress.
+ (BOOL)clearCache:(NSError * __autoreleasing *)error
{
__block BOOL successOnLegacyPath = NO;
__block NSError *errorOnLegacyPath = nil;
NSString *cacheDirPath = [NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask, YES) lastObject];
NSArray *allSubPaths = [[NSFileManager defaultManager] subpathsAtPath:cacheDirPath];
if (!allSubPaths) {
NSLog(#"No subpaths of cache:\n%#", cacheDirPath);
} else {
[allSubPaths enumerateObjectsUsingBlock:^(NSString *subpath, NSUInteger idx, BOOL *stop) {
static NSString * const kNSURLSessionPathComponent = #"nsurlsession"; // this is a non-documented way, Uncle Apple can change the path at any time
if ([subpath containsString:kNSURLSessionPathComponent]) {
successOnLegacyPath = [[NSFileManager defaultManager] removeItemAtPath:[cacheDirPath stringByAppendingPathComponent:subpath]
error:&errorOnLegacyPath];
if (!successOnLegacyPath) {
NSLog(#"Error while deleting cache subpath:\n%#\nError:\n%#", subpath, errorOnLegacyPath);
}
// First we find is the root > bail out
*stop = YES;
}
}];
}
if (!successOnLegacyPath && !errorOnLegacyPath) {
// Couldn't find the nsurlsession's cache directory
if (error) *error = [NSError errorWithDomain:NSCocoaErrorDomain
code:NSFileNoSuchFileError
userInfo:nil];
// OR
successOnLegacyPath = YES;
}
return successOnLegacyPath;
}
This is not a solution and this is recommended to use if no download is in progress. Haven't tested what's happening if there are running downloads and trying to delete the tmp files.
Even if I find a solution, the previously created tmp files still remain untracked so those need to be deleted by a method like this.
Btw, this seems to be the same question - without conclusion.

iOS Downloading multiple files in the background

I have read few articles but could not find what I was looking for so here's my query.
I am downloading few files from the server and there is a case where the user locks his screen, in this case the ios device loses its network connectivity and the file sync fails.
I have read few article on NSURLSession but it is available from iOS 7 onwards and the app I am working on supports from iOS 6 and later.
So is there a way where I can download 20 or 30 files in the background or when the user hits the lock screen in a generic fashion without having to worry about which OS version I am supporting.
As of now I have read that we have 30 seconds to perform network activity so is there a limitation to the number of server calls in these 30 seconds?
About my code, I am having a class named as DownloadFiles which calls a service and the service returns me an array of fileURL and using NSData I am fetching these files and saving them in the doc directory, so while implementing the background call thing do I need to pass the index of my array which will detect the current file which is downloading and then carry on from the next index.
for(NSDictionary *dict in filearray) {
NSString *fileURL = [[dict valueForKey:#"FileURL"]stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding];
NSData *fileData = [NSData dataWithContentsOfURL:theFileURL];
if (fileData.length==0 || fileData==nil || theFileURL==nil) {
NSLog(#"empty file URL = %#",theFileURL);
}
if (fileData.length!=0){
BOOL savefile = [fileData writeToFile:[HTML_SERVER_FILES stringByAppendingPathComponent:[dict valueForKey:#"FileName"]] atomically:YES];
if (savefile!=YES) {
NSLog(#"Not saved file = %#",theFileURL);
}else{
NSLog(#"file saved at path %#",HTML_SERVER_FILES);
}
fileData = nil;
}
}
Please let me know what needs to be done in this case.

How to know when iTunes has finished copying a file to the documents directory [duplicate]

I'm working on iOS project that supports iTunes file sharing feature. The goal is realtime tracking incoming/changed data's.
I'm using (kinda modified) DirectoryWatcher class from Apple's sample code
and also tried this source code.
The data is NSBundle (*.bundle) and some bundles are in 100-500 MB ranges, depends on its content, some video/audio stuff. The bundles has xml based descriptor file in it.
The problem is any of these codes above fires notification or whatever else when the data just started copying and but not when the copy/change/remove process finished completely.
Tried next:
checking file attributes:
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
BOOL fileBusy = [[fileAttrs objectForKey:NSFileBusy] boolValue];
looking for the fileSize changes:
dispatch_async(_checkQueue, ^{
for (NSURL *contURL in tempBundleURLs) {
NSInteger lastSize = 0;
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
NSInteger fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
do {
lastSize = fileSize;
[NSThread sleepForTimeInterval:1];
fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
NSLog(#"doing job");
} while (lastSize != fileSize);
NSLog(#"next job");
}
);
any other solutions?
The solution above works great for bin files, but not for .bundle (as .bundle files are directory actually). In order to make it work with .bundle, you should iterate each file inside .bundle
You can use GCD's dispatch sources mechanism - using it you can observe particular system events (in your case, this is vnode type events, since you're working with file system).
To setup observer for particular directory, i used code like this:
- (dispatch_source_t) fileSystemDispatchSourceAtPath:(NSString*) path
{
int fileDescr = open([path fileSystemRepresentation], O_EVTONLY);// observe file system events for particular path - you can pass here Documents directory path
//observer queue is my private dispatch_queue_t object
dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_VNODE, fileDescr, DISPATCH_VNODE_ATTRIB| DISPATCH_VNODE_WRITE|DISPATCH_VNODE_LINK|DISPATCH_VNODE_EXTEND, observerQueue);// create dispatch_source object to observe vnode events
dispatch_source_set_registration_handler(source, ^{
NSLog(#"registered for observation");
//event handler is called each time file system event of selected type (DISPATCH_VNODE_*) has occurred
dispatch_source_set_event_handler(source, ^{
dispatch_source_vnode_flags_t flags = dispatch_source_get_data(source);//obtain flags
NSLog(#"%lu",flags);
if(flags & DISPATCH_VNODE_WRITE)//flag is set to DISPATCH_VNODE_WRITE every time data is appended to file
{
NSLog(#"DISPATCH_VNODE_WRITE");
NSDictionary* dict = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:nil];
float size = [[dict valueForKey:NSFileSize] floatValue];
NSLog(#"%f",size);
}
if(flags & DISPATCH_VNODE_ATTRIB)//this flag is passed when file is completely written.
{
NSLog(#"DISPATCH_VNODE_ATTRIB");
dispatch_source_cancel(source);
}
if(flags & DISPATCH_VNODE_LINK)
{
NSLog(#"DISPATCH_VNODE_LINK");
}
if(flags & DISPATCH_VNODE_EXTEND)
{
NSLog(#"DISPATCH_VNODE_EXTEND");
}
NSLog(#"file = %#",path);
NSLog(#"\n\n");
});
dispatch_source_set_cancel_handler(source, ^{
close(fileDescr);
});
});
//we have to resume dispatch_objects
dispatch_resume(source);
return source;
}
I found two rather reliable (i.e. not 100% reliable but reliable enough for my needs) approaches, which only work in conjunction with polling the contents of the directory:
Check NSURLContentModificationDateKey. While the file is being transferred, this value is set to the current date. After transfer has finished, it is set to the value the original file had: BOOL busy = (-1.0 * [modDate timeintervalSinceNow]) < pollInterval;
Check NSURLThumbnailDictionaryKey. While the file is being transferred, this value is nil, afterwards it cointains a thumbnail, but probably only for file types from which the system can produce a thumbnail. Not a problem for me cause I only care about images and videos, but maybe for you. While this is more reliable than solution 1, it hammers the CPU quite a bit and may even cause your app to get killed if you have a lot of files in the import directory.
Dispatch sources and polling can be combined, i.e. when a dispatch source detects a change, start polling until no busy files are left.

Can't write large file (> 200MB, 700MB in my case) in ios device?

I am having the problem of writing large file (>200 MB) in iOS device (iPad) but in the simulator it works perfect.
I am using NSFileManager to create file and NSFileData to write file.
I think there is no problem in my code as it runs fine in the simulator.
Does anyone having the same problem?
To elaborate my situation:
I am saving chunk of files (3MB each) in my device which works fine. That means for a 300 MB file I have 100 chunks. Now, from the 100 chunks I want to create the actual file. So I am using the NSFileManager to create file in first iteration and then using NSFileData to write the 3MB data at the end of the file. While running the program it crashes after 61 chunks. I am guessing there might have some memory related issues in the iPad.
I am saving the chunk of files in fileDir in the format data-0, data-1, data-2...
I am applying the decrypt operation on data but for simplicity I have removed that portion.
// List of chunk files
NSArray *filelist= [[NSFileManager defaultManager] contentsOfDirectoryAtPath:fileDir error:err];
for(int i = 0; i < [filelist count]; i++) {
// Read the chunk of file
fileName = [[NSString alloc] initWithFormat:#"data-%d", i];
filePath = [fileDir stringByAppendingPathComponent:fileName];
fileReadHandle = [NSFileHandle fileHandleForReadingAtPath:filePath];
// Write in tempFile
if(offset == 0){
if([[NSFileManager defaultManager] createFileAtPath:tempFile contents:data attributes:nil]){
fileWriteHandle = [NSFileHandle fileHandleForWritingAtPath:tempFile];
NSLog(#"File was created!");
} else {
NSLog(#"File was not created.");
}
} else {
[fileWriteHandle seekToEndOfFile]; // Tried with comment out this line but same problem
// Write the decrypted data from chunk
[fileWriteHandle writeData:[[fileReadHandle readDataToEndOfFile] decryptedAES256DataUsingKey:AESEncryptionKey error:err]];
}
}
Edit (11.02.2013)
I tried with my previous code where I omitted the data decryption part.
Interestingly, the problem was in the decryption part I guess cause without the decryption it works fine. I have added the decryption code. For decryption I am using NSData+CommonCrypto library (it's non ARC) but my project is in ARC.
It could be an operating system issue because the NSFileHandle is never being closed for each chunk. I would recommend closing it.
Also, it looks like you have your variables declared outside the scope of the for loop. Unless you need those variables outside the loop, it's generally good to keep the scope of your variables as small as possible, especially if you are using ARC and are trying to think about when memory will be released.
If you think that the NSFileHandle is holding onto data in memory, try to use the -synchronizeFile method after writing each chunk to make sure in memory changes are reflected to disk.
Also, I moved the creation of the file you are writing to outside the loop, because it's easier to follow for me.
Try this adjustment:
// List of chunk files
NSArray *filelist= [[NSFileManager defaultManager] contentsOfDirectoryAtPath:fileDir error:err];
if([[NSFileManager defaultManager] createFileAtPath:tempFile contents:[NSData data] attributes:nil]){
NSLog(#"File was created!");
} else {
NSLog(#"File was not created.");
}
NSFileHandle *fileWriteHandle = [NSFileHandle fileHandleForWritingAtPath:tempFile];
for(int i = 0; i < [filelist count]; i++) {
// Read the chunk of file
NSString *fileName = [NSString stringWithFormat:#"data-%d", i];
NSString *filePath = [fileDir stringByAppendingPathComponent:fileName];
NSFileHandle *fileReadHandle = [NSFileHandle fileHandleForReadingAtPath:filePath];
NSData *data = [fileReadHandle readDataToEndOfFile];
// No longer using the file
[fileReadHandle closeFile];
// Write in tempFile
[fileWriteHandle writeData:data];
[fileWriteHandle synchronizeFile];// Flush any data in memory to disk
}
[fileWriteHandle closeFile];
Modifying the following code worked like a magic,
#autoreleasepool {
[fileWriteHandle writeData:[[fileReadHandle readDataToEndOfFile] decryptedAES256DataUsingKey:AESEncryptionKey error:err]];
}

Resources