I'm making an iOS app that decodes an h264 stream using video-toolbox. I create the stream with ffmpeg on a PC and send it to an iPhone using RTP. It's working nicely when I use this command to create it:
ffmpeg -y -f:v rawvideo -c:v rawvideo -s 1280x720 -pix_fmt bgra -r 30 -an -i - -pix_fmt yuv420p -c:v libx264 -tune zerolatency -preset fast -b:v 5M -refs 1 -g 30 -profile:v high -level 4.1 -f rtp rtp://192.168.1.100:5678
The iPhone receives and displays all the frames. However, when I enable intra-refresh
-intra-refresh 1
decoding fails with error code -12909 (-8969 on simulator) when VTDecompressionSessionDecodeFrame() is called.
I take care of processing UDP packets to extract NAL Units, so I triple checked this process and discarded a problem with this part of the code.
I didn't find any info about Video-toolbox not supporting intra-refresh, so the question is, does Video-toolbox support intra-refresh? and if it does, am I missing something in the ffmpeg side that makes the stream not supported by Video-toolbox?
Do I have to add something to the CMVideoFormatDescriptionRef apart from creating it with SPS and PPS data using CMVideoFormatDescriptionCreateFromH264ParameterSets()?
Yes, Video-toolbox supports intra-refresh
No, nothing to do with ffmpeg
No, don't need to do anything special with CMVideoFormatDescriptionRef
I figured it out, I was creating a new VTDecompressionSession each time I was receiving SPS and PPS NALUs, so the decoder was loosing the context.
It was working without intra-refresh because in that case a complete I-Frame is received right after SPS and PPS, so it doesn't need context from previous frames.
With intra-refresh enabled, only the first frame is a complete I-Frame, then the decoder relies on context from previous frames and must use the same VTDecompressionSession.
Related
I'm building an iOS app where re-encoding and trimming a video in the background is necessary.
I can not use iOS libraries (AVFoundation) since they rely on the GPU and no app can access the GPU if it's backgrounded.
Due to this issue I switched to FFMpeg and compiled it (alongside libx264) and integrated it on my iOS app.
To sum things up what I need is:
Trim the video for the first 10 seconds
re-scale the video
After a couple of weeks - and banging my head against the wall quite often - I managed to:
split the video container into streams (demuxing)
copy the audio stream into the output stream (no decoding or encoding)
decode the video stream, run the necessary filters per frame, encode each resulting frame and remux it to the output stream (I decode the h264, filter it, re-encode it back to h264)
If I were to run ffmpeg through the command line I would run it like this:
ffmpeg -i input.MOV -ss 0 -t 10 -vf scale=320:240 -c:v libx264 -preset ultrafast -c:a copy output.mkv
My concern is how to trim the video? Although I could count the number of video frames that I encode/decode and based on the FPS decide when to stop I cannot do the same with the audio since I'm only demuxing and remuxing it.
Ideally - before scaling the video - I would run a process to trim the video by copying the 10 seconds of each stream (video and audio) into a new video container.
How to I achieve this through the AV libraries?
I know you can do this with one call to ffmpeg:
ffmpeg -i input.MOV -filter_complex [0:v]trim=duration=10.0,scale=320:240[vid];[0:a]atrim=duration=10.0[aud] -map [vid] -map [aud] -c:v libx264 -preset ultrafast -c:a libvo_aacenc -b:a 128k -flags +aic+mv4 output.mkv
I am making a site in Ruby in which I have a series of images, (almost like a powerpoint) and I need to automatically convert those images into one continuous video file (mov, mpeg) that shows each image for 5 seconds or so. Any one have any clues where to start.
I'm also open to using another language if there are tools to get the job done.
You could probably use FFmpeg to do this. Here's an example from the FFmpeg Wiki on the subject:
ffmpeg -framerate 1/5 -i img%03d.jpg -c:v libx264 -r 30 -pix_fmt yuv420p -movflags +faststart out.mp4
What this would do is...
-i img%03d.jpg
Read input from a series of JPEG files named img001.jpg, img002.jpg and so on
-framerate 1/5
...at one frame per five seconds...
-c:v libx264
...then turn it into H.264/MPEG-4 AVC...
-r 30
...at thirty frames per second...
-pix_fmt yuv420p
...with YUV420 pixel format (really, all the FFmpeg flags work here)...
-movflags +faststart
...after encoding completes, relocate some data to the beginning of the file so playback can begin before the file is completely downloaded...
out.mp4
...and store it into out.mp4.
If you were using this from Ruby you'd likely launch a subprocess. The flags would be similar if you really want a (QuickTime) .mov file instead of H.264 MPEG-4.
I'm developing upload video (taken from iPhone) to my server.
However, I have no idea how to implement.
Any source code objective-c or swift will be welcomed.
I have 120fps or 240fps video (It's a slo-mo).
When I playback these video on my iPhone6. I can see slo-mo effect.
(I know playback frame rate is 30fps.)
I want to convert that video before upload to my server, from 120/240 fps to 30fps video. (I mean not adjusting playback frame rate, it means video transcode to 30fps.)
Additionally, I want to check slo-mo effect start-point and end-point.
(Maybe iPhone record this information to video binary(it might be reside in file's header.)
Well, I guess if I use ffmpeg library, it should be easy(?).
So any suggestions will be welcomed.
Here are ffmpeg command lines I use to import into Adobe Premiere:
Video:
ffmpeg -i <input MOV> -filter "setpts=4.0*PTS" -r 30 -an videofilename.mp4
4.0 in -filter means that the iphone slomo video was shot at 120fps, i.e. 4 × 30fps; the related -r 30 parameter is for 30fps. For example if you want to export as 60fps, use setpts="2.0*PTS" -r 60
-an discards the audio stream
Audio:
ffmpeg -i <input MOV> -filter "setpts=4.0*PTS" audiofilename.mp3
At this point you have the video and audio streams in separate files. You can probably use ffmpeg to recombine them.
But there's a catch: the iPhone will record the audio stream at normal speed, meaning the converted sound track will be 4 times (in my example) shorter than the converted video track. if you use Premiere, import both video and audio files, right click on the sound track in your timeline, choose "Speed/Duration", and set speed=25% (or 50% for 120fps to 60fps)
For 120fps footage with 44100 audio sample rate:
ffmpeg -i in.MOV -filter_complex "[0:v]setpts=4.0*PTS[v];[0:a]asetrate=11025,aresample=44100[a]" \
-map "[v]" -map "[a]" -r 30 out.mp4
For 240fps footage with 44100 audio sample rate:
ffmpeg -i in.MOV -filter_complex "[0:v]setpts=8.0*PTS[v];[0:a]asetrate=5512.5,aresample=44100[a]" \
-map "[v]" -map "[a]" -r 30 out.mp4
For 240fps footage with 48000 audio sample rate:
ffmpeg -i in.MOV -filter_complex "[0:v]setpts=8.0*PTS[v];[0:a]asetrate=6000,aresample=48000[a]" \
-map "[v]" -map "[a]" -r 30 out.mp4
The quality of the resulting video will be low. Increasing quality is a science in itself https://trac.ffmpeg.org/wiki/Encode/H.264 currently the -crf parameter (with a low number in the 0–63 range) seems to be the simplest way to increase quality (at the price of file size and encoding time). For example, use -crf 18 before out.mp4
Based on FFMPEG documentation: https://trac.ffmpeg.org/wiki/How%20to%20speed%20up%20/%20slow%20down%20a%20video
With the help of https://superuser.com/questions/292833/how-to-change-audio-frequency for the audio slowdown (chaining atempo=2.0,atempo=2.0 gives a horrible-sounding result).
I use ffmpeg to encode my sample videos following the recommanded bitrates in Technical Note TN2224, then use HLS tools to segment it and create playlists, finally create the variant plist file "all.m3u8"
I used the validation tool to validate my HLS content, it ended up showing except for the 64k audio only bandwidth is low, others are stay in the same bandwidth, I opened "all.m3u8" using text editor and seeing that all other bitrate contents are using the same bandwidth. No matter how I change parameters in the ffmpeg command, I still can't correct them. The following command is the one I used to encode contents:
ffmpeg -i input.m4v -acodec libfaac -vcodec libx264 -s 480x360 -b 350k -r 29.97 -vpre medium output.mp4
The following command is for generating the segments and plists:mediafilesegmenter -b http://www.example.com/stream/ -I -f ~/Documents/sample/ output.mp4
The following command is for generating the all.m3u8:variantplaylistcreator -o all.m3u8 http://www.example.com/stream/110/prog_index.m3u8 ~/Documents/sample/110/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/110/iframe_index.m3u8 http://www.example.com/stream/200/prog_index.m3u8 ~/Documents/sample/200/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/200/iframe_index.m3u8 http://www.example.com/stream/350/prog_index.m3u8 ~/Documents/sample/350/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/350/iframe_index.m3u8 http://www.example.com/stream/550/prog_index.m3u8 ~/Documents/sample/550/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/550/iframe_index.m3u8 http://www.example.com/stream/64/prog_index.m3u8 ~/Documents/sample/64/prog_index.m3u8
and in my "all.m3u8", the bandwidths are all 523894:
Please allow me to ask two more basic questions:
In the tech note, recommanded bitrates are 64 Kbps, 110 Kbps, 200 Kbps, 350 Kbps, 550 Kbps, I wonder if this value includes the audio bitrate or exclude the audio.
How do you insert keyframe to segment? Because in the document it says:"You must include at least one keyframe per segment, preferably more. If you only include one, put it at the beginning of the segment." I don't quite get how you can do it.
Thank you very much for your help and I do appreciate your time.
Jason,
To create all.m3u8 should it not be given multiple m3u8 files each corresponding to a different bitrate?
I am guessing you run ffmpeg say 4 times to create for 4 bitrate files. Then you run the segmenter 4 times to create 4 set of segments and its individual m3u8 files.
Finally you have to tell the variantplaylistcreator where the location of the various m3u8 files per bitrate to create a single master m3u8 file.
Eg.
variantplaylistcreator -o mymedia_all.m3u8 http://mywebserver/mymedia_lo/prog_index.m3u8 mymedia_lo.plist http://mywebserver/mymedia_med/prog_index.m3u8 mymedia_med.plist http://mywebserver/mymedia_hi/prog_index.m3u8 mymedia_hi.plist
I don't see you providing the various filese seperately. I hope you get the picture.
EDIT: To answer your other questions:
Bitrates include audio. What you need to do is ensure you have a fixed key frame interval in your encoding. This will allow the segmenter to segment the files at regular intervals. you don't insert anything anywhere.
Out of curiosity why not directly use ffmpeg to give you the output segmented files? It supports it.
Thanks for everybody's attention and suggestions. I finally figured it out. The reason why the bandwidth stayed the same for different bitrate is that my ffmpeg command missed couple settings. I ended up using the following command:ffmpeg -i inputVideo.m4v -f mpegts -acodec libfaac -ar 44100 -ab 64k -vcodec libx264 -b 350k -s 480x360 -r 29.97 -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 200k -maxrate 350k -bufsize 350k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 4:3 -g 30 -async 2 output.ts
I put it here so that other people who have the same problem as me will have a reference.
It sounds like you may have uncovered a bug in variantplaylistcreator. I recommend to verify that the sub-streams really are the bitrate you expect, and if it's really putting the wrong value, to report it to apple.
It might have something to do with using multiple -iframe-url. I can't understand why it would be necessary to specify it more than once. Adaptive streaming won't work if the substreams have different I-frame positions -- at least all of the segment boundaries must be aligned.
If you need to fix the playlist up programmatically, I recommend to use ffprobe (from ffmpeg suite) to extract the bitrate of each substream, and replace the bandwidth number with the extracted value.
Has anyone tried this ?
What's the best practice for this?
FMS live streams are using the RTMP protocol:
ffmpeg -i rtmp://***server/path* **-acodec copy -vcodec copy -y *captured***.flv**
Here, we are saving the whole stream to an FLV file, which is Flash's static movie file format and so can always preserve all RTMP audio and video codecs without conversion.
You can then extract any frames you want, e.g.
ffmpeg -i *captured***.flv -s** starttime -vframes 1 -f image2 -vcodec mjpeg *captured***.jpg**
If you are ambitious and know exactly what time offsets and intervals you want to capture in advance, you can do both steps at once, e.g. one frame every second:
ffmpeg -i rtmp://***server/path* **-r 1 -f image2 -vcodec mjpeg *captured***%d.jpg**
All commandlines have not been tested, will need fixing but give you a good impression