I'm looking to use an existing video player library for iOS apps with HLS support so that I can implement a player with some very specific networking behavior, as opposed to letting Apple decide the size and timing of requests. It needs to be customizable enough to support new networking policies such that I can override the request sizes, change what files are requested, and read data from an existing local cache file. In short, I'm attempting to override the networking portion that actually calls out and fetches the segments such that I can feed in data from partial cache as well as make specific algorithmic changes to the timing and size of external HTTP requests.
I've tried AV Foundation's AVAssetResourceLoaderDelegate Protocol but, unless there is something I'm not seeing, there doesn't seem to be a way to override the outgoing requests and just feed bytes to the media player.
I've also looking into VLC but unfortunately my current project is incompatible with a GPL license.
Ideally, there would be a way to directly feed bytes or complete segments to MPMoviePlayerController, but I can't find any way of accomplishing this in the API. The only method I'm aware that works is using a local HTTP server, which I have been doing but it seems overly complicated when all I'm really trying to do is override some internal networking code.
Any suggestions of another way of doing this?
Related
I'm looking for a way to implement real-time streaming of video (and optionally audio) from iOS device to a browser. In this case iOS device is a server and browser is a client.
Video resolution must be in the range 800x600-1920x1080. Probably the most important criteria is lag that should be less than 500 msec.
I've tried a few approaches so far.
1. HLS
Server: Objective-C, AVFoundation, UIKit, custom HTTP-server implementation
Client: JS, VIDEO tag
Works well. Streams smoothly. The VIDEO tag in the browser handles incoming video steam out of the box. This is great! However, it has lags that are hard to minimize. It feels like this protocol was built for non-interactive video streaming. Something like twitch where a few seconds of lag is fine.
Tried Enabling Low-Latency. A lot of requests. A lot of hassle with the playlist. Let me know if this is the right option and I have to push harder in this direction.
2. Compress every frame into JPEG and send to a browser via WebSockets
Server: Objective-C, AVFoundation, UIKit, custom HTTP-server implementation, WebSockets server
Client: JS, rendering via IMG tag
Works super-fast and super-smooth. Latency is 20-30 msec! However, when I receive a frame in a browser, I have to load it using loading from a Blob field via base64 encoded URL. At the start, all of this works fast and smoothly, but after a while, the browser starts to slow down and lags. Not sure why I haven't investigated too deeply yet. Another issue is that frames compressed as JPEGs are much larger (60-120kb per frame) than MP4 video stream of HLS. This means that more data is pumped through WiFi, and other WiFi consumers are starting to struggle. This approach works but doesn't feel like a perfect solution.
Any ideas or hints (frameworks, protocols, libraries, approaches, e.t.c.) are appreciated!
HLS
… It feels like this protocol was built for non-interactive video streaming …
Yep, that's right. The whole point of HLS was to utilize generic HTTP servers as media streaming infrastructure, rather than using proprietary streaming servers. As you've seen, several tradeoffs are made. The biggest problem is that media is chunked, which naturally causes latency of at least the size of the chunk. In practice, it ends up being the size of a couple chunks.
"Low latency" HLS is a hack to return to the methods we had before HLS, with servers that just stream content from the origin, in a way compatible with all the HLS stuff we have to deal with now.
Compress every frame into JPEG and send to a browser via WebSockets
In this case, you've essentially recreated a video codec, and added the overhead of Web Sockets. Also, with the base64 encoding rather than sending it binary, you're adding extra CPU and memory requirements, as well as ~33% overhead in bandwidth.
If you really wanted to go this route, you could simply use MediaRecorder, an HTTP PUT request, stream the output of the recorder, send it to the server, to relay on to the client over HTTP. The client then just needs a <video> tag referencing some URL on the server, and nothing special to playback. You'll get nice low latency without all the overhead and hassle.
However, don't go that route. Suppose the bandwidth drops out? What if some packets are lost and you need to re-sync? How will you set up communication between each end to continually adjust quality, buffering, codec negotiation, etc.? What if peer-to-peer connections are advantageous?
Use WebRTC
It's a full purpose-built stack for maintaining low latency. Libraries are available for most any stack on most any platform. It works in browsers.
Rather than reinventing all of this, you can take advantage of what's there.
The downside is complexity... it isn't easy to get started with, but well worth it for most low latency use cases.
What I'm doing
Basically I'm writing simple a Q&A site with an option to create links to specific positions in media files. As of now the app is intended to be used in LAN environment only.
I have put a video in appRoot/public folder and created a view using
html5 video tag.
It works and even seeking is available. Wow...
What I don't understand
I'm clueless as to the tech behind and its limitations.
It just worked, so I don't even know a key word to hit google with.
What I know
With the way I'm doing:
No encryption
No way to prevent users to save video files
No automatic trans-coding available
The real question
What is the name of the tech behind.
How well can rails handle streaming and seeking requests with the way I did as compared to using dedicated video streaming servers or gems.
As long as your underlying web server understands how to handle the MIME types for video, and responds correctly to byte range requests - as it seems to be - that's all you need. The underlying mechanics of streaming video with HTML5 is that the browser asks for a chunk of content as a range of bytes from the source (enough to keep the buffer full) and the server delivers it.
You might want to look at using ffmpeg to optimize your videos so that the metadata is in the right place in the file to start streaming quicker.
You've correctly pointed out the limitations of the solution in your environment. The other thing to be aware of is capacity - if the videos are long and a lot of people are accessing them concurrently then without caching (in a LAN via a caching proxy or on the internet via a CDN service) your server capacity may be stretched
Bear with me on this one please.
When setting response of a WinJS.xhr response I can set it to, among other things, to 'ms-stream' or blob. I was hoping to leverage the stream concept when downloading a file in such a way that I don't have to keep the whole response in memory (video files can be huge).
However, all I can do with 'ms-stream' object is read it with an MSStreamReader. This would be great if I could say to it 'consume 1024 bytes from the stream, and 'loop' this, until stream is exhausted. However from reading the docs (haven't tried this, so correct me if I'm wrong), it appears I can only read from the stream once (e.g. readAsBlob method) and I can't set the start position. This means I need to read the whole response into memory as a blob. Which I can achieve with responseType set to 'blob' in the first place. So what is the point of MSStream anyway?
Well, it turns out that the method msDetachStream gives access to underlying stream and doesn't interrupt the download process. I initially thought that any data that was not downloaded was lost when calling this since the docs mention that MSStream object is closed.
I wrote a blog post a while back to help answer questions about MSStream and other oddball object types that you encounter in WinRT and the host for JavaScript apps. See http://www.kraigbrockschmidt.com/2013/03/22/msstream-blob-objects-html5/. Yes, you can use MSStreamReader to for some work (it's a synchronous API), but you can also pass an MSStream to URL.createObjectURL to assign it to an img.src and so forth.
With MSStream, here's some of what I wrote: "MSStream is technically an extension of this HTML5 File API that provides interop with WinRT. When you get MSStream (or Blob) objects from some HTML5 API (like an XmlHttpRequest with responseType of “ms-stream,” as you’d use when downloading a file or video, or from the canvas’ msToBlob method), you can pass those results to various WinRT APIs that accept IInputStream or IRandomAccessStream as input. To use the canvas example, the msRandomAccessStream in a blob from msToBlob can be fed into APIs in Windows.Graphics.Imaging for transform or transcoding. A video stream can be similarly worked with using the APIs in Windows.Media.Transcoding. You might also just want to write the contents of a stream to a StorageFile (that isn’t necessarily on the file system) or copy them to a buffer for encryption."
So MSStreamReader isn't the end-all. The real use of MSStream is to pass the object into WinRT APIs that accept the aforementioned interface types, which opens many possibilities.
Admittedly, this is an under-documented area, which is exactly why I wrote my series of posts under the title, Q&A on Files, Streams, Buffers, and Blobs (the initial post is on http://www.kraigbrockschmidt.com/2013/03/18/why-doesnt-storagefile-close-method/).
Is there any way, using currently available SDK frameworks on Cocoa (touch) to create a streaming solution where I would host my mp4 content on some server and stream it to my iOS client app?
I know how to write such a client, but it's a bit confusing on server side.
AFAIK cloudKit is not suitable for that task because behind the scenes it keeps a synced local copy of datastore which is NOT what I want. I want to store media content remotely and stream it to the client so that it does not takes precious space on a poor 16 GB iPad mini.
Can I accomplish that server solution using Objective-C / Cocoa Touch at all?
Should I instead resort to Azure and C#?
It's not 100% clear why would you do anything like that?
If you have control over the server side, why don't you just set up a basic HTTP server, and on client side use AVPlayer to fetch the mp4 and play it back to the user? It is very simple. A basic apache setup would do the job.
If it is live media content you want to stream, then it is worth to read this guide as well:
https://developer.apple.com/Library/ios/documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/StreamingMediaGuide.pdf
Edited after your comment:
If you would like to use AVPlayer as a player, then I think those two things don't fit that well. AVPlayer needs to buffer different ranges ahead (for some container formats the second/third request is reading the end of the stream). As far as I can see CKFetchRecordsOperation (which you would use to fetch the content from the server) is not capable of seeking in the stream.
If you have your private player which doesn't require seeking, then you might be able to use CKFetchRecordsOperation's perRecordProgressBlock to feed your player with data.
Yes, you could do that with CloudKit. First, it is not true that CloudKit keeps a local copy of the data. It is up to you what you do with the downloaded data. There isn't even any caching in CloudKit.
To do what you want to do, assuming the content is shared between users, you could upload it to CloudKit in the public database of your app. I think you could do this with the CloudKit web interface, but otherwise you could create a simple Mac app to manage the uploads.
The client app could then download the files. It couldn't stream them though, as far as I know. It would have to download all the files.
If you want a streaming solution, you would probably have to figure out how to split the files into small chunks, and recombine them on the client app.
I'm not sure whether this document is up-to-date, but there is paragraph "Requirements for Apps" which demands using HTTP Live Streaming if you deliver any video exceeding 10min. or 5MB.
A potential client has come to me asking for a an app which will stream a six hour audio file. The user needs to be able to set the "playback head" to any position along the file. Presumably, this means that the app must not be forced to download the entire file before it beings playing back starting at an arbitrary
An added complication -- there are actually four files which need to be streamed and mixed simultaneously.
My questions are:
1) Is there an out-of-the box technology which will allow me random access of streaming audio, on iOS? Can this be done with standard server technology and a single long file, or will it involve some fancy server tech?
2) Which iOS framework is best suited for this. Is there anything high-level that would allow me to easily mix these four audio files?
3) Can this be done entirely with standard browser technology on the client side? (i.e. HTML5)
Have a close look at the MP3 format. It is remarkably easy and efficient to parse, chop up into little bits, and reassemble into a custom stream.
Hence rolling your own server-side code to grab what you want and send to the client will not be as crazy or difficult as it may sound.
MP3 is also widely supported by various clients. I strongly suspect any HTML5 capable browser will be able of play the stream you generate via a long-lived bit-rate regulated HTTP request.