I am looking at using this file upload process in my asp.net MVC website. http://www.codeproject.com/Articles/830704/Gigabit-File-uploads-Over-HTTP?fid=1870992
It's working great so far. I can see the file being uploaded in chunks (on the server). Essentially they are placed in my upload directory in a directory that has the file name. In that directory, I see the chunks, such as "LargeFile.00000001.csv.tmp", "LargeFile.000000002.csv.tmp", etc....
If the upload is canceled I can still see the chunks on the server. Question: How can I "resume" the upload later on?
I could look at the folder name and file chunk names and decipher where I left off.
Related
I am implementing a resumable upload protocol that uploads in the background on iOS, which means I have to use an NSURLSessionUploadTask with a file. Since it's a resumable upload protocol, the file needs to be truncated based on the data that has already been received by the server, so I need to save a new temporary file to disk that has only the bytes to be uploaded within it.
If I can create that temporary upload file in the tmp/ or /Library/Caches/, can I trust that it will be kept as long as the NSURLSession is running?
EDIT: When an upload fails, the server will be saving the bytes it has already received and communicating that to the client. The client then should only send part of the file, which is why I need to create a smaller temporary file that must not be deleted mid-upload.
Huh? You provide the entire file, and the system takes care of managing the partial upload/download, and notifies you once the transfer is complete. In the case of a download, t hands you a temporary file once the download is complete and you have to save it to a permanent location.
You should not be mucking around with partial files at all.
EDIT:
You don't have access to tmp or /Library/Caches/, except through the sandbox. You can get access to the caches directory with the call
[NSSearchPathForDirectoriesInDomains(
NSCachesDirectory,
NSUserDomainMask, YES) lastObject];
It's my understanding that the caches directory only gets purged on restart, or if the device gets critically low on space, but I seem to remember that the docs are vague on when, exactly, the caches directory gets cleared.
You would probably be better off saving your file to the documents directory, then deleting it once you're done with it.
The answer to your question is no. NSURLSessionUploadTask's description appears to support keeping the source file around but it's misleading:
"In iOS, when you create an upload task for a file in a background session, the system copies that file to a temporary location and streams data from there"
But it says nothing about whether it will keep the original source file in the tmp directory. Specifically for your case where your server supports uploading partial files and you need to restart them after failures. Or in the more common situation where you need to manually restart an entire failed upload, for example from a retry-able server error, or if user killed your app and then restarted it (iOS doesn't continue uploads for user killed apps).
In these cases you can't count on the file still being around if you create it in the apps tmp directory. The file system programming guide tells us this.
https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html
"Use this directory to write temporary files that do not need to persist between launches of your app. Your app should remove files from this directory when they are no longer needed; however, the system may purge this directory when your app is not running. The contents of this directory are not backed up by iTunes or iCloud."
So any tmp directory files can be deleted by iOS when your app stops running, and I can confirm I've seen this in production releases of our app. If you think you may need the source file for the upload again, you must store it in your own app directory, and manage deleting it yourself when done with it. Sorry, extra work, but I don't know of any way around it.
I have a program that uploads files to a ftp server. If upload is complete then I delete the files. Occasionally the files are not deleted and because the upload is automated I get an infinite loop where 100 copies of the file reach the server. If I try to manually delete the file from explorer I'm able to do it but for some reason the Deletefile command from my app doesn't work.
I've tried raising last OSError and I get nothing.
The files are stored on a mapped drive. Is there any workaround? If I upload 30 files to a ftp server sometimes one or two files cannot be deleted after loading them to the server.
Is there any way to close the file even if it is opened from another program? Something like unlocker does?
How can i see if my application is locking the file? I haven't implemented any locking mechanism inside the app.
Thanks.
I'm working with Apple's iTunes EPF data files. I'll daily need to download, unzip and then process 1-3GB of data in .tbz files every day.
I've got a Rails app, hosted on Heroku, with most asset storage being taken care of on S3.
But what I'm having trouble with is the flow for getting the EPF files from Apple.
There are 3 files I'll be downloading. Each are .tbz files varying in size from 1GB to down to ~20MB.
Heroku doesn't have a way to reliable store files, so I assume I need to download the files directly to S3? Then would I somehow unzip them there?
That's where I'm hitting a snag. I know how to actually get the files from Apple and on to S3, but decompressing them is where I'm not following.
And since the data files can be pretty large, minimizing the transfer over S3 is critical to keeping costs down.
Is there a service that can let me download the Apple files to their servers, decompress, and then upload to S3 the necessary files?
Heroku's file system is ephemeral, but you can still write out to /tmp as a temporary scratch space to download, unzip, do whatever processing you need, re-package (if needed), and then uploaded to S3. Because of automatic dyno restarts (or manual restarts), just make sure your service knows how to gracefully resume if interrupted.
I have a website where i upload a zip file and then the serverside decompresses it. i've since moved to amazons S3 service which does not allow such things as decompressing.
I'm wondering, is there a way to check or monitor the status of that zip file- and then run my model/method for pushing to s3? i'd like to run it immediately after it's decompressed- otherwise i'd try a cronjob or something.
The only conclusion i can think of right now is to output the files unzipped in my view. then selecting those files and submitting again to the method for uploading. but this seems cumbersome.
any thoughts on this?
I am trying to upload a zip file of 350mb - 500mb to server. It gives "ENOSPC" error.
Is it possible to upload file in chunks and receive it on server as one file ?
or
Use custom location for tmpfs, so that it will be independent of system tmp, because in my case tmp is of 128MB only.
Why not use the Web-server uploading feature like nginx-upload and apache-upload
Not sure what it is called in apache but I guess apache too has it
if you are using Nginx
there is also a nginx-upload-progress which can be helpful if you want to track the progress of the upload
Hope this help