HTTP LIve Streaming - ios

Ok, I have been trying to wrap my head around this http live streaming. I just do not understand it and yes I have read all the apple docs and watched the wwdc videos, but still super confused, so please help a wanna be programer out!!!
The code you write goes on the server? not in xcode?
If I am right how do i set this up?
Do I need to set up something special on my server? like php or something?
How do use the tools that are supplied by Apple.. segmenter and such?
Please help me,
Thanks

HTTP Live Streaming
HTTP Live Streaming is a streaming standard proposed by Apple. See the latest draft standard.
Files involved are
.m4a for audio (if you want a stream of audio only).
.ts for video. This is a MPEG-2 transport, usually with a h.264/AAC payload. It contains 10 seconds of video and it is created by splitting your original video file, or by converting live video.
.m3u8 for the playlist. This is a UTF-8 version of the WinAmp format.
Even when it's called live streaming, usually there is a delay of one minute or so during which the video is converted, the ts and m3u8 files written, and your client refresh the m3u8 file.
All these files are static files on your server. But in live events, more .ts files are added, and the m3u8 file is updated.
Since you tagged this question iOS it is relevant to mention related App Store rules:
You can only use progressive download for videos smaller than 10 minutes or 5 MB every 5 minutes. Otherwise you must use HTTP Live Streaming.
If you use HTTP Live Streaming you must provide at least one stream at 64 Kbps or lower bandwidth (the low-bandwidth stream may be audio-only or audio with a still image).
Example
Get the streaming tools
To download the HTTP Live Streaming Tools do this:
Get a Mac or iPhone developer account.
Go to https://developer.apple.com and search for "HTTP Live Streaming Tools", or look around at https://developer.apple.com/streaming/.
Command line tools installed:
/usr/bin/mediastreamsegmenter
/usr/bin/mediafilesegmenter
/usr/bin/variantplaylistcreator
/usr/bin/mediastreamvalidator
/usr/bin/id3taggenerator
Descriptions from the man page:
Media Stream Segmenter: Create segments from MPEG-2 Transport streams for HTTP Live Streaming.
Media File Segmenter: Create segments for HTTP Live Streaming from media files.
Variant Playlist Creator: Create playlist for stream switching from HTTP Live streaming segments created by mediafilesegmenter.
Media Stream Validator: Validates HTTP Live Streaming streams and servers.
ID3 Tag Generator: Create ID3 tags.
Create the video
Install Macports, go to the terminal and sudo port install ffmpeg. Then convert the video to transport stream (.ts) using this FFMpeg script:
# bitrate, width, and height, you may want to change this
BR=512k
WIDTH=432
HEIGHT=240
input=${1}
# strip off the file extension
output=$(echo ${input} | sed 's/\..*//' )
# works for most videos
ffmpeg -y -i ${input} -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -s ${WIDTH}x${HEIGHT} -vcodec libx264 -b ${BR} -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 7 -trellis 0 -refs 0 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 200k -maxrate ${BR} -bufsize ${BR} -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 30 -qmax 51 -qdiff 4 -level 30 -aspect ${WIDTH}:${HEIGHT} -g 30 -async 2 ${output}-iphone.ts
This will generate one .ts file. Now we need to split the files in segments and create a playlist containing all those files. We can use Apple's mediafilesegmenter for this:
mediafilesegmenter -t 10 myvideo-iphone.ts
This will generate one .ts file for each 10 seconds of the video plus a .m3u8 file pointing to all of them.
Setup a web server
To play a .m3u8 on iOS we point to the file with mobile safari.
Of course, first we need to put them on a web server. For Safari (or other player) to recognize the ts files, we need to add its MIME types. In Apache:
AddType application/x-mpegURL m3u8
AddType video/MP2T ts
In lighttpd:
mimetype.assign = ( ".m3u8" => "application/x-mpegURL", ".ts" => "video/MP2T" )
To link this from a web page:
<html><head>
<meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
</head><body>
<video width="320" height="240" src="stream.m3u8" />
</body></html>
To detect the device orientation see Detect and Set the iPhone & iPad's Viewport Orientation Using JavaScript, CSS and Meta Tags.
More stuff you can do is create different bitrate versions of the video, embed metadata to read it while playing as notifications, and of course have fun programming with the MoviePlayerController and AVPlayer.

This might help in swift:
import UIKit
import MediaPlayer
class ViewController: UIViewController {
var streamPlayer : MPMoviePlayerController = MPMoviePlayerController(contentURL: NSURL(string:"http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/sl.m3u8"))
override func viewDidLoad() {
super.viewDidLoad()
streamPlayer.view.frame = self.view.bounds
self.view.addSubview(streamPlayer.view)
streamPlayer.fullscreen = true
// Play the movie!
streamPlayer.play()
}
}
MPMoviePlayerController is deprecated from iOS 9 onwards. We can use AVPlayerViewController() or AVPlayer for the purpose. Have a look:
import AVKit
import AVFoundation
import UIKit
AVPlayerViewController :
override func viewDidAppear(animated: Bool){
let videoURL = NSURL(string: "https://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4")
let player = AVPlayer(URL: videoURL!)
let playerViewController = AVPlayerViewController()
playerViewController.player = player
self.presentViewController(playerViewController, animated: true) {
playerViewController.player!.play()
}
}
AVPlayer :
override func viewDidAppear(animated: Bool){
let videoURL = NSURL(string: "https://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4")
let player = AVPlayer(URL: videoURL!)
let playerLayer = AVPlayerLayer(player: player)
playerLayer.frame = self.view.bounds
self.view.layer.addSublayer(playerLayer)
player.play()
}

Another explanation from Cloudinary http://cloudinary.com/documentation/video_manipulation_and_delivery#http_live_streaming_hls
HTTP Live Streaming (also known as HLS) is an HTTP-based media streaming communications protocol that provides mechanisms that are scalable and adaptable to different networks. HLS works by breaking down a video file into a sequence of small HTTP-based file downloads, with each download loading one short chunk of a video file.
As the video stream is played, the client player can select from a number of different alternate video streams containing the same material encoded at a variety of data rates, allowing the streaming session to adapt to the available data rate with high quality playback on networks with high bandwidth and low quality playback on networks where the bandwidth is reduced.
At the start of the streaming session, the client software downloads a master M3U8 playlist file containing the metadata for the various sub-streams which are available. The client software then decides what to download from the media files available, based on predefined factors such as device type, resolution, data rate, size, etc.

Related

Can Google's Speech API accept an external Video URL?

I recently figured out that Google's Vision API can accept an external image URL and I was curious if anyone knew if Google's Speech could accept an external video URL such as a YouTube video?
The code I have in my mind would look something like this:
def transcribe_gcs(yotube_url):
"""Asynchronously transcribes the audio file specified by the gcs_uri."""
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
client = speech.SpeechClient()
audio = types.RecognitionAudio(uri=youtube_url) # swapped out gcs_uri with youtube_url
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.FLAC,
# sample_rate_hertz=16000,
language_code='en-US')
operation = client.long_running_recognize(config, audio)
print('Waiting for operation to complete...')
response = operation.result(timeout=90)
# Each result is for a consecutive portion of the audio. Iterate through
# them to get the transcripts for the entire audio file.
for result in response.results:
# The first alternative is the most likely one for this portion.
print(u'Transcript: {}'.format(result.alternatives[0].transcript))
print('Confidence: {}'.format(result.alternatives[0].confidence))
I was curious if anyone knew if Google's Speech could accept an
external video URL such as a YouTube video?
It needs to be a local path to your audio file (less than 1 min audio file) or GCS URI for audio file longer than 1 minute. What you're thinking is not possible, the audio/video file needs to be in GCS.
I think you can achieve this by streaming same video (for example on wowza or on any server of your choice.) and then simply extract audio using lets say ffmpeg and pass this to google. It should work. use StreamingRecognizeRequest instead of RecognitionAudio.

how to set video quality for ios 270 360 480 720 1080

To set video quality for ios.
I have tried to load m3u8 video url from server and i downloaded the m3u8 file & i segregate all RESOLUTION from video quality & AFTER SEGMENTS get the bandwidth of url in array.
When i load base url sample.m3u8 it has video & audio after that i set the base url of before segments and i have append the bandwidth url from array it was loading video as per quality selected but no audio came.
To achieve this i have made some solutions will work
I make separate to run original url which contains both video & audio and i run separately low bandwidth url which contains no audio to make sync
ex: RESOLUTION=1280x720,SAMPLE_720p_v4.m3u8
SAMPLE.m3u8
#EXTM3U
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-0",NAME="Default",AUTOSELECT=YES,DEFAULT=YES,URI="segments/SAMPLE_audio_v4.m3u8"
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=30681000,CODECS="avc1.640028",URI="segments/SAMPLE_1080p_iframe.m3u8"
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=30140000,CODECS="avc1.4d001f",URI="segments/SAMPLE_720p_iframe.m3u8"
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=15431000,CODECS="avc1.42001f",URI="segments/SAMPLE_480p_iframe.m3u8"
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=11009000,CODECS="avc1.42001e",URI="segments/SAMPLE_360p_iframe.m3u8"
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=7850000,CODECS="avc1.420015",URI="segments/SAMPLE_270p_iframe.m3u8"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4080000,RESOLUTION=1280x720,CODECS="avc1.640028,mp4a.40.2",AUDIO="audio-0"
segments/SAMPLE_1080p_v4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3471000,RESOLUTION=1280x720,CODECS="avc1.4d001f,mp4a.40.2",AUDIO="audio-0"
segments/SAMPLE_720p_v4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1934000,RESOLUTION=854x480,CODECS="avc1.42001f,mp4a.40.2",AUDIO="audio-0"
segments/SAMPLE_480p_v4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1106000,RESOLUTION=640x360,CODECS="avc1.42001e,mp4a.40.2",AUDIO="audio-0"
segments/SAMPLE_360p_v4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=837000,RESOLUTION=480x270,CODECS="avc1.420015,mp4a.40.2",AUDIO="audio-0"
segments/SAMPLE_270p_v4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=185000,CODECS="mp4a.40.2",AUDIO="audio-0"
segments/SAMPLE_audio_v4.m3u8
Use the preferredPeakBitRate property on your playeritem https://developer.apple.com/documentation/avfoundation/avplayeritem/1388541-preferredpeakbitrate you need to pass a valid bandwidth value.
Not sure why you are downloading the m3u8 file AVFoundation manage this for you.

Looping an AVPlayer video stream and refreshing the bitrate on each loop

I'm looping my streamed videos (not live stream) via .m3u8 playlist and each time the video restarts, it plays the video with the same bitrate adapting that occurs the first time you watch the video (bad quality -> good quality). Is there a way to refresh the stream quality each time the video loops so that the beginning gets replaced with the higher-rate bitrate seamlessly? Instead of just re-playing what was initially loaded?
Apple's AVPlayer attempts to load the first stream listed in the HLS playlist. So if you want the highest quality stream to be loaded first by default, you need to specify it as the first stream in the playlist file.
With that in mind, one way of achieving what you need to achieve is to have a different m3u8 file for each of your streams.
For example, if you have a three variant stream playlist, you would have three .m3u8 playlists.
Then in your view controller where you are using your AVPlayer, you need to keep a reference to the last observed bitrate and most recent bit rate:
var lastObservedBitrate: double = 0
var mostRecentBitrate: double = 0
You would then need to register a notification observer on your player with notification name: AVPlayerItemNewAccessLogEntryNotification
NSNotificationCenter.defaultCenter().addObserver(self, selector:#selector(MyViewController.accessEventLog(_:)), name: AVPlayerItemNewAccessLogEntryNotification, object: nil)
Whenever the access log is updated, you can then inspect the bitrate and stream used using the following code:
func accessLogEvent(notification: NSNotification) {
guard let item = notification.object as? AVPlayerItem,
accessLog = item.accessLog() else {
return
}
accessLog.events.forEach { lastEvent in
let bitrate = lastEvent.indicatedBitrate
lastObservedBitrate = lastEvent.observedBitrate
if let mostRecentBitrate = self.mostRecentBitrate where bitrate != mostRecentBitrate {
self.mostRecentBitrate = bitrate
}
}
}
Whenever your player loops, you can load the appropriate m3u8 file based on your lastObservedBitrate. So if your lastObservedBitrate is 2500 kbps, you would load your m3u8 file that has the 2500kbps stream at the top of the file.
Shameless plug: We've designed something similar in our video api. All you need to do is request the m3u8 file with your connection type: wifi or cellular and lastObservedBitrate and our API will vend you the best possible stream for that bitrate, but still have the ability to downgrade/upgrade the stream if network conditions change.
If you are interested in checking it out visit: https://api.storie.com or https://github.com/Storie/StorieCloudSDK

HLS stream not working on Apple devices

I have a live RTSP stream that i have managed to transcode to HLS via VLC. Now it works perfect on Android and on desktop browsers (via flash).
But not on Apple (i can test it on iPad and desktop Safari on my virtual machine). I can see the player but when i press the 'play' button all i see is a black rectangle inside the player. On desktop Safari there is also a text 'Loading...' near the play/pause button and nothing else happens.
My HTML:
<video id="player" controls style="width:100%; height:100%">
<source src="http://178.79.164.114/playlist.m3u8" type="application/x-mpegURL">
</video>
The command for vlc:
vlc -I dummy rtsp://<stream-url> --sout '#transcode{width=320,height=240,fps=25,vcodec=h264,vb=256,acodec=none,venc=x264{aud,profile=baseline,level=30,keyint=30,bframes=0,ref=1,nocabac}}:std{access=livehttp{seglen=10,delsegs=true,numsegs=5,index=/path/to/server/directory/playlist.m3u8,index-url=http://178.79.164.114/seg-########.ts},mux=ts{use-key-frames},dst=/path/to/server/directory/seg-########.ts}'
And an example of the playlist file:
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:NO
#EXT-X-MEDIA-SEQUENCE:179
#EXTINF:9.60,
http://178.79.164.114/seg-00000179.ts
#EXTINF:9.60,
http://178.79.164.114/seg-00000180.ts
#EXTINF:9.60,
http://178.79.164.114/seg-00000181.ts
#EXTINF:9.61,
http://178.79.164.114/seg-00000182.ts
#EXTINF:9.59,
http://178.79.164.114/seg-00000183.ts
And here is the strange output of ffprobe http://178.79.164.114/playlist.m3u8 (why there are these N/A and the variant_bitrate is 0?). Maybe it can help:
Input #0, hls,applehttp, from 'http://178.79.164.114/playlist.m3u8':
Duration: N/A, start: 3995.330722, bitrate: N/A
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p, 320x240 [SAR 11:12 DAR 11:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
I have also configured correct MIME types for .m3u8 and .ts files and spent a day searching and trying different options for the transcode command: width, height, bitrate, fps, different profiles and levels... - nothing works. But if i try some examples from apple (http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8) - all is fine, though it's not a live stream.
If anyone has any ideas or has a possibility to test my stream with the mediastreamvalidator - please help.
UPDATE
Now i'm experimenting with variant playlist but it changes nothing.
The player might expect muxed video and audio so add a silent audio track.
The Apple HLS documentation says:
The media segment files are normally produced by the stream segmenter, based on input from the encoder, and consist of a series of .ts files containing segments of an MPEG-2 Transport Stream carrying H.264 video and AAC, MP3, or AC-3 audio
Support for audio-only streams is mentioned in Technical Note TN2224 and the 7th revision of the protocol introduced support for alternate renditions (unmuxed streams) but this is done with EXT-X-MEDIA tags in a master playlist controlling the playback (yours is a media playlist).

Encoding SWF to video with Melt

I'm doing a project which requires converting SWF movies to H.264 video on server-side, to be able to play them both in Flash player and on iPhone/iPad. And I really got stuck.
I'm using Melt from http://www.mltframework.org/ and this is my command-line:
melt movie.swf -consumer avformat:video.mp4 r=30 s=640x360 f=mp4 acodec=aac ab=128k ar=48000 vcodec=libx264 b=1000k an=1
It does play in Flash player, but fails to play on iDevices. I googled for iPhone video requirements and it seems my video files do satisfy them(frame size, framerate and bitrate). What settings should I change to make it play?
I've spent a lot of time in google but managed to gather all the pieces, so these are parameters that work for iPhone:
r=30 s=640x360 f=mp4 acodec=aac ab=128k ar=48000 vcodec=libx264 level=30 b=1024k flags=+loop+mv4 cmp=256 partitions=+parti4x4+parti8x8+partp4x4+partp8x8+partb8x8 me_method=hex subq=7 trellis=1 refs=1 bf=0 flags2=+mixed_refs-wpred-dct8x8 coder=0 wpredp=0 me_range=16 g=250 keyint_min=25 sc_threshold=40 i_qfactor=0.71 qmin=10 qmax=51 qdiff=4 maxrate=10M bufsize=10M an=1 threads=0
Also, I use faac -w to convert audio to appropriate format and MP4Box to join video and sound.

Resources