I want to compare two different file uploading /downloading application.
Basically the scenarios would be, I want to measure the time taken by each application for uploading a file/downloading a file.
One way of doing that is using a stop watch and monitoring the upload and download.
I am looking for a tool which would record the time taken when the upload /download completes.
Related
I have to load a lot of data onto an iOS device with 128GB storage, for use by my app. The data is around 2,000 files of around 40Mb each, total is around 80GB - 100Gb.
I control the iOS device and the load machine/program and the local network they're on, and time is not critically important (if it takes a week to load, that's OK). I can format the data as required to facilitate the load.
I've done some iOS programming, but I'm not sure where to start looking for a solution to this. If you can outline the broad approach to use and which iOS docs to read up on, that's all I need.
Hoping for a solution where I can format the data and write the program, plug the iPad in to the Mac and say 'start loading' and come back when it's done.
We discovered that it is possible to load data one device at a time using iTunes, but that isn't a good answer for us.
We added a 'load data' button into the application. When triggered it loads a configuration file from a hard coded local network address, then retrieves all the data files listed in the configuration file using REST GETs from the local server, storing them into the application /Documents directory.
This is a good approach for us; it is secure, allows multiple devices to load data at the same time, and doesn't need any manual file loading : start the app, press 'load', and wait for it to finish.
Scenario:
As a user I am able to take (an unlimited amount of) photos and videos which are stored in the apps documents folder. Each of these media files gets a record within a Sqlite database with additional information (for exeample a caption). All this is possible to do completely offline.
Back online I get a dialog with a list of all the videos and photos I took and a button which starts an upload process.
Each file is uploaded after the other together with its metadata by making a multipart POST request to the server. The response of the server is stored together with the metadata in the Sqlite database (so there is no fire and forget).
Reliable solutions?
If I am reading and understanding this chart correctly, the most simple solution would be to wrap each of these uploads within a Task. Side effect: after 10 minutes every task would be cancelled, which becomes a problem by having a slow connection or very large files (for example a very long video).
The recommended way would be to use NSUrlSession/Background transfer service.
Which leads me to my question:
Is it possible to wrap multipart POSTs in NSURLSessionDataTasks and would this be reliable, even if the task is running longer than 10 minutes or the user is suspending the app?
As I am a Xamarin/C# guy I would really appreciate some sample snippets for a working multipart upload, even if it's in Objective-C ;-).
Almost and... yes.
Background Transfer service works with NSUrlSessionDownloadTasks and NSUrlSessionUploadTasks only. Not NSUrlSessionDataTasks, as described here.
Other than this "basic" limitation, it's safe to use background transfer service with upload tasks.
The 10-minute-freepass-in-the-background no longer applies on iOS 7 (basically, it's there, but different), however, with NSURLSession and background transfer service you do not need it.
I've a blog post here for background transfer service, based on download tasks.
An important thing to note is that, starting a task basically means that it will actually start sometime and actually finish some other time. This depends on whether the device is on cellular or Wi-Fi and other factors which are (probably) only known to iOS (and Apple).
It is possible to implement a feature that allows users to watch videos as they are uploaded to server by others. Is html 5 suitable for this task? But flash? Are there any read to go solutions, don't want to reinvent the wheel. The application will be hosted on a dedicated server.
Thanks.
Of course it is possible, the data is there isnt it?
However it will be very hard to implement.
Also I am not so into python and I am not aware of a library or service suiting your requirements, but I can cover the basics of video streaming.
I assume you are talking about video files that are uploaded and not streams. Because, for that, there are obviously thousands of solutions out there...
In the most simple case the video being uploaded is already ready to be served to your clients and has a so called "faststart atom". They are container format specific and there are sometimes a bunch of them. The most common is the moov-atom. It contains a lot of data and is very complex, however in our use case, in a nutshell, it holds the data that enables the client to begin playing the video right away using the data available from the beginning.
You need that if you have progressive download videos (youtube...), meaning where a file is served from a Webserver. You obviously have not downloaded the full file and the player already can start playing.
If the fastastart atom was not present, that would not be possible.
Sometimes it is, but the player for example cannot display a progress bar, because it doesnt know how long the file is.
Having that covered the file could be uploaded. You will need an upload solution that writes the data directly to a buffer or a file. (file will be easier...).
This is almost always the case, for example PHP creates a file in the tmp_dir. You can also specify it if you want to find the video while its being uploaded.
Well, now you can start reading that file byte by byte and print that data to a connection to another client. Just be sure not to go ahead of what has already been recieved and written. You would probaby initiate your upload with a metadata set in memory that holds the current recieved byte position and location of the file.
Anyone who requests the file after the uploaded has started can just recieve the entire file, or if the upload is not yet finished, get it from your application.
You will have to throttle the data delivery or pause it when the data becomes short. This will appear to the client almost as a "slow connection". However you will have to echo some data from time to time to prevent the connection from closing. But if your upload doesnt stall, and why shoud it?, that shouldnt be a problem.
Now if you want to have someting like on the fly transcoding of various input formats into your desired output format, things get interesting.
AFAIK ffmpeg has neat apis which lets you directly deal with datasterams.
Also handbrake is a very good tool, however you would need to take the long road using external executeables.
I am not really aware of your requirements, however if your clients are already tuned in, for example on a red 5 streaming server, feeding data into a stream should also work fine.
Yes, take a look at Qik, http://qik.com/
"Instant Video Sharing ... Videos can be viewed live (right as they are being recorded) or anytime later."
Qik provides developer APIs, including ones like these:
qik.stream.subscribe_public_recent -- Subscribe to the videos (live and recorded)
qik.user.following -- Provides the list of people the user is following
qik.stream.public_info -- Get public information for a specific video
It is most certainly to do this, but it won't be trivial. And no, I don't think that you will find an "out of the box" solution that will require little effort on your behalf.
You say you want to let:
users watch videos as they are uploaded to server by others
Well, this could be interpreted two different ways:
Do you mean that you don't want a user to have to refresh the page before seeing new videos that other users have just finished uploading?
Or do you mean that you want one user to be able to watch a partially uploaded video (aka another user is still in the process of uploading it and right now the server only contains a partial upload of the video)?
Implementing #1 wouldn't be hard at all whatsoever. You would just need an AJAX script to check for newly uploaded videos, and those videos could then be served to the user in whatever way you choose. HTML5 vs. Flash isn't really a consideration here.
The second scenario, on the other hand, would require quite a bit of effort. I am guessing that HTML5 might not be mature enough to handle this type of situation. If you are not looking
to reinvent the wheel and don't have a lot of time to dedicate to this feature than I would say that you would be out of luck. You may be able to use ffmpeg to parse partial video files and feed them to a Flash player, but I would think of this as a large task.
Problem statement:
I want to upload a large binary (such as an audio clip) from an iOS app to S3, and I'd like to make the app's handling of disconnects (or low connectivity) as robust as possible, preferably by uploading the binary as a series of chunks.
Unfortunately, neither the AWSiOS SDK, nor ASI's S3 framework seem to support to multi-part uploads, or indicate that they plan to add support. I realize that I can initiate a 'longish' upload using beginBackgroundTaskWithExpirationHandler: and that'll give me a window of time to complete the upload (currently 600 seconds, I believe), but what's to be done if I'm not in a situation to complete said upload within that timeframe?
Aside from worrying about completing tasks within that time frame, is their a 'best practice' for how an app should resume uploads, or even just break a larger upload into smaller chunks?
I've thought about writing a library to talk to S3's REST API specifically for multi-part uploads, but this seems like a problem other have either been solved, or realized needn't be solved (perhaps for being completely in appropriate for the platform).
Another (overly complicated) solution would be chunking the file on the device, uploading those to S3 (or elsewhere) and have them re-assembled on S3 via a server process. This seems even more unpalatable than rolling my own library for multi-part upload.
How are others handling this problem?
Apparently I was looking at some badly out of date documentation.
in AmazonS3Client see:
- (S3MultipartUpload * AmazonS3Client)initiateMultipartUploadWithKey:(NSString *)theKey withBucket:(NSString *)theBucket
Which will give you a S3MultipartUpload which will contain an uploadId.
You can then put together an S3UploadPartRequest using initWithMultipartUpload: (S3MultipartUpload *) multipartUpload and send that as you usually would.
S3UploadPartRequest contains an int property partNumber where you can specify the part # you're uploading.
you can write some code to do so, you can refer code from http://dextercoder.blogspot.in/2012/02/multipart-upload-to-amazon-s3-in-three.html. Core java code, steps can be used for iOS.
Do large file uploads block an applications request/response cycle? I have an app that allows users to upload multiple large files (images in particular). These files are stored on a remote host. I cannot use async background jobs to upload these images as these have to be immediately accessible to the user once the upload finishes. How best should i handle such large uploads? Does it affect concurrency? This is my first time with uploads on a large scale. What should i be wary of other than the huge bills of course? Any input from developers who have created apps which use large file uploads will be greatly appreciated.
Why can't you use an async upload, and just handle the event that signifies that it's done? That's generally how async operations work - you kick them off and then store the pointer somewhere, and then either handle the "Complete" event, or just periodically iterate through he pointers for uploads you've started and check each one to see if it's complete.
It's an old question but still, I was worried about the same problem with large files uploads thinking that the processes get blocked while the file is getting uploaded but it turned out, if I got it right, that the nginx and probably other servers as well buffer the content of the file while its being sent so no rails processes get blocked, only when the upload is finished and rails is processing it, like resizing images or something.