I need to decode mp4 file and draw it using OpenGL in ios app. I need to extract and decode h264 frames from mp4 file and I heard what it posible to do using CoreMedia. Anybody has any idea how to do it? Any examples of CoreMedia using?
It's not Core Media you're looking for, it's AVFoundation. In particular, you'd use an AVAssetReader to load from your movie and iterate through the frames. You then can upload these frames as OpenGL ES textures either by using glTexImage2D() or (on iOS 5.0) by using the much faster texture caches.
If you don't want to roll your own implementation of this, I have working AVFoundation-based movie loading and processing via OpenGL ES within my GPUImage framework. The GPUImageMovie class encapsulates movie reading and the process of uploading to a texture. If you want to extract that texture for use in your own scene, you can chain a GPUImageTextureOutput to it. Examples of both of these classes can be found in the SimpleVideoFileFilter and CubeExample sample applications within the framework distribution.
You can use this directly, or just look at the code I wrote to perform these same actions within the GPUImageMovie class.
Related
I am using AVFoundation to re-export a video. I am reading it in using AVAssetReader, and then exporting again using AVAssetWriter. Works great for "normal" videos.
However, I've noticed that for videos where everything is constant except for a small region (e.g: https://www.youtube.com/watch?v=VyhG-AKLK-Y) it doesn't compress very well.
The reason is probably that the encoder has no way of knowing the contents of the video so it uses a generic compression method.
I came across this: https://developer.apple.com/videos/play/wwdc2014/513/ and implemented the steps needed to support multi-pass encoding (uses an additional pass so that it can analyze the contents of the video first).
The results, while better than the default, doesn't match up to ffmpeg or videos exported from premiere (much worse quality with higher filesize).
Does anyone have any insights on how to best optimize for videos like this?
I am working on a video editing application.
I need a function to do chroma keying and replace green background in a video file (not live feed) with an image or another video.
I have looked through the GPUImage framework but it is not suitable for my project as I cannot use third party frameworks for this project so I am wondering if there is another way to accomplish this.
Here are my questions:
Does it need to be done through Shaders in Opengl es?
Is there another way to access the frames and replace the background doing chroma keying using the AV Foundation framework?
I am not too versed in Graphics processing so I really appreciate any help.
Apple's image and video processing framework is CoreImage. There is sample code for Core Image which includes a ChromeKey example.
No it's not in Swift, which you asked for. But as #MoDJ says; video software is really complex, reuse what already exists.
I would use Apple's Obj-c sample code to get something that works then, if you feel you MUST have it in Swift, port it little by little and if you have specific problems ask them here.
I'm trying to decode frames from multiple video files, and use them as opengl texture.
I know how to decode a h264 file using AVAssetReader object, but it seems you have to read the frames after you call startReading in a while loop when the status is AVAssetReaderStatusReading. What I want to do is to call startReading then call copyNextSampleBuffer anywhere anytime I want. In this way, I can create a new video reader class from AVAssetReader, and load video frames from multiple video files whenever I want to use them as opengl textures.
Is this doable?
Short answer is yes, you can decode one frame at a time. You will need to manage the decode logic yourself and the most simple thing is to just allocate a buffer of BGRA pixels and then copy the framebuffer data into your temp buffer. Be warned that you will likely not be able to find a little code snippit that does all this. Thing is, streaming all the data from movies into OpenGL is not easy to implement. I would suggest that you avoid attempting to do this yourself and use a 3rd party library that already implements the hard stuff. If you want to see a complete example of something like this already implemented then you can have a look at my blog post Load OpenGL textures with alpha channel on iOS. This post shows how to stream video into OpenGL but you would need to decode from h.264 to disk first using this approach. It should also be possible to use other libraries to do the same thing, just keep in mind that playing multiple videos at the same time is resource intensive, so you may run into the limits of what can be done on your hardware device quickly. Also, if you do not actually need OpenGL textures, then it is a lot easier to just operate on CoreGraphics APIs directly under iOS.
I am trying to crop videos both taken in my app and uploaded from the user's photo library. I am looking to crop every video to be the size of the iPhone 5s screen (I know that sounds dumb, but that's what I need to do).
Can I do this using the AV Foundation framework or do I need to use Core Video? I've made multiple attempts with AV Foundation and gotten nowhere.
Also if you could link to any helpful tutorials or code samples that would be greatly appreciated.
I'm using Objective-C and working on an app designated for iOS 7+.
Thanks!
1) Use AVAssetReader to open your video and extract CMSampleBuffers
Link
2) Modify CMSampleBuffer:
Link
3) Create AVAssetWriter and add modified CMSampleBuffers to it's input
Link
In this article CVPixelBuffer is used as an input to AVAssetWriter
using adapter. You actually don't need an adapter because
you have CMSampleBuffers ready you can add them straight
to the input using appendSampleBuffer: method.
I've been exploring options on iOS to achieve hardware accelerated decoding of raw H.264 stream and so far I only found that the only option is to write the H.264 stream into an MP4 file and then pass the file to an instance of AVAssetReader. Although this method works, it's not particulary suitable for realtime applications. AVFoundation reference indicates the existence of a CALayer that can display compressed video frames (AVSampleBufferDisplayLayer) and I believe this would be a valid alternative to the method mentioned above. Unfortunately this layer is only available on OSX. I would like to file an enchament radar but before I do so I would like to know from someone that has experience with this layer if indeed could be use to display H.264 raw data if was available on iOS. Currently in my app, the decompressed YUV frames are rendered via openGLES. Using this layer means that I will not need to use openGLES anymore?
In iOS 8 the AVSampleBufferDisplayLayer class is available now.
Take a Look and have Fun