I had no luck getting h264 videos with RGBA pixels to work on iOS (tested on iOS 10.2) Is it possible? The Apple docs doesn't say much about it: https://developer.apple.com/library/content/documentation/Miscellaneous/Conceptual/iPhoneOSTechOverview/MediaLayer/MediaLayer.html
I don't have much interesting code to share since it's just that the AVPlayer doesn't display videos with RGBA pixels.
It's not possible to decode H264 into exactly 'RGBA' however:
a AVPlayerItemVideoOutput can be set to:kCVPixelFormatType_32BGRA using the kCVPixelBufferPixelFormatTypeKey
and
VTDecompressionSessionCreate also allows : kCVPixelFormatType_32BGRA.
Then when rendering I swizzle the pixels like this: gl_FragColor.bgra = texture2D(SamplerRGB, v_textureCoordinate);
So the answer is yes but you have to do the rendering and swizzle.
/A
(Edit. added links to code):
Here is a really great project by Apple that will get you going.. Real-time Video Processing Using AVPlayerItemVideoOutput
Just change it from YUV to RGB and bypass the 'colorConversionMatrix' part of the shader.
Good luck!
Related
What's the common modern standard for animated video overlays? (e.g. if you want to add an animated logo to video recorded from the camera)
During research, I've found the following options:
GIF - seems to be pretty outdated technology
FLV - supports alpha-channel, but no longer supported by Adobe.
Requires FFMPEG.
PNG sequence - the downside of this is having multiple files for each
frame.
What's the right format/technology to use?
Ideally, what is natively supported on iOS (doesn't require FFMPEG)?
If you want to overlay your custom video animation over video which user will be recorded I suggest to use GPUImage framework which allow a lot video/photo customization's and different graphic effects. For example how to mix two videos: nice article. Also I suggest you to read article about Chroma key which are something like standard of video/photo mixing. (because as I understand you just want make something like watermark?). GPUImage also has Chroma key filter which you can use in your purpose.
By default Apple supports h264 codec in mp4 container. So your video should be in this codec.
Hope I fully answered on your question
The best way to add overlays using the AVFoundation framework supplied by apple itself. Speaking about the other ways such as GIF, FLV, they are not supported natively by APPLE which puts you out of luck.
Apple suggests various tools such as AVVideoCompositionCoreAnimationTool that lets you stitch the Core Animations and the videos together.
Here is a link that explains how to add various effects such as
Colored borders with custom sizes.
Multiple overlays.
Text for subtitles or captions.
Tilt effects.
Twinkle, rotate, and fade animation effects!
I am not sure how much of this is application for the application that wanted to add animations while recording. May be some one else could help in it. I hope this helps you about the native way to add animations in recorded videos in iOS.
I'm trying to decode a video in real time (30 fps) and display /modify it with OpenGL. On an iPod touch, if I decode a video that I took with the camera, decoding a frame can take over 1s, while 30 fps should be 0.03s max. Thus the result is not very good..
Is it possible to achieve that with AVAssetReader ? For example Instagram applies filters (I think GLSL shaders) in real time on a video, and they can even navigate in the video. Instagram works fine on the ipod touch.
The code to decode can be found in the answer here :
Best way to access all movie frames in iOS
And more specifically here : Hardware accelerated h.264 decoding to texture, overlay or similar in iOS
Thank you in advance
Due to the very limited information you provided, I have to assume that your video sequence are compressed in the format of YUV and you set settings of AVAssetReader with other format like kCVPixelFormatType_32BGRA which forces iOS use hardware acceleration to convert colour space for you, then you feel it slowly. I suggest that no settings to set, just use its original pixel format.
Actually my app was just doing too much work on the CPU, I had another process analyzing images. When I removed it, the decoding was really fast.
This question is about iOS. On Android, it is very easy to use OpenGL ES 2.0 to render a texture on a view (for previewing) or to send it to an encoder (for file writing). I haven't been able to find any tutorial on iOS to achieve video playback (previewing video effect from a file) and video recording (saving a video with an effect) with shader effects. Is this something possible with iOS?
I've come across a demo about shaders called GLCameraRipple but I have no clue about how to use it more generically. Ex: With AVFoundation.
[EDIT]
I trampled on this tutorial about OpenGL ES, AVFoundation and video merging on iOS while searching for a snippet. That's another interesting entry door.
It's all very low-level stuff over in iOS land, with a whole bunch of pieces to connect.
The main thing you're likely to be interested in is CVOpenGLESTextureCache. As the CV prefix implies, it's part of Core Video, in this case its primary point of interest is CVOpenGLESTextureCacheCreateTextureFromImage which "creates a live binding between the image buffer and the underlying texture object". The documentation further provides you with explicit advice on use of such an image as a GL_COLOR_ATTACHMENT — i.e. the texture ID returned is usable both as a source and as a destination for OpenGL.
The bound image buffer will be tied to a CVImageBuffer, one type of which is a CVPixelBuffer. You can supply pixel buffers to an AVAssetWriterInputPixelBufferAdaptor wired to an AVAssetWriter in order to output to a video.
In the other direction, an AVAssetReaderOutput attached to a AVAssetReader will vend CMSampleBuffers which can be queried for attached image buffers (if you've got video coming in and not just audio, there'll be some) that can then be mapped into OpenGL via a texture cache.
In my app I have to play an alpha channel video as an overlay over the current view (I'm planning to achieve this alpha channel video using GPUImageAlphaBlendFilter or GPUImageChromaKeyBlendFilter), so I wanted to know if the output video after applying these filters can be played using GPUImage? If we can, then can I get some sample code for the same.
I know AVAnimator is an option but I want to apply filters to these overlay videos i.e.brightness,saturation etc which has to be visible while video is being played because of which I can't use AVAnimator. But this being the next step for now I want to know how to play video using GPUImage.
Thanks in advance! :]
Well, even though I like telling people about AVAnimator, Brad Larson's GPUImage is specifically designed to be a GPU based filtering framework for iOS apps. Application of real time effects like the ones you describe is exactly what GPUImage was designed for. See GPUImage Chroma Key filter
I'm trying to figure out the best approach to display a video on a GL texture while preserving the transparency of the alpha channel.
Information about video as GL texture is here: Is it possible using video as texture for GL in iOS? and iOS4: how do I use video file as an OpenGL texture?.
Using ffmpeg to help with alpha transparency, but not app store friendly is here:
iPhone: Display a semi-transparent video on top of a UIView?
The video source would be filmed in front of a green screen for chroma keying. The video could be untouched to leave the green screen or processed in a video editing suite and exported to Quicktime Animation or Apple Pro Res 4444 with Alpha.
There are multiple approaches that I think could potentially work, but I haven't found a full solution.
Realtime threshold processing of the video looking for green to remove
Figure out how to use the above mentioned Quicktime codecs to preserve the alpha channel
Blending two videos together: 1) Main video with RGB 2) separate video with alpha mask
I would love to get your thoughts on the best approach for iOS and OpenGL ES 2.0
Thanks.
The easiest way to do chroma keying for simple blending of a movie and another scene would be to use the GPUImageChromaKeyBlendFilter from my GPUImage framework. You can supply the movie source as a GPUImageMovie, and then blend that with your background content. The chroma key filter allows you to specify a color, a proximity to that color, and a smoothness of blending to use in the replacement operation. All of this is GPU-accelerated via tuned shaders.
Images, movies, and the live cameras can be used as sources, but if you wish to render this with OpenGL ES content behind your movie, I'd recommend rendering your OpenGL ES content to a texture-backed FBO and passing that texture in via a GPUImageTextureInput.
You could possibly use this to output a texture containing your movie frames with the keyed color replaced by a constant color with a 0 alpha channel, as well. This texture could be extracted using a GPUImageTextureOutput for later use in your OpenGL ES scene.
Apple showed a sample app at WWDC in 2011 called ChromaKey that shows how to handle frames of video passed to an OpenGL texture, manipulated, and optionally written out to a video file.
(In a very performant way)
It's written to use a feed from the video camera, and uses a very crude chromakey algorithm.
As the other poster said, you'll probably want to skip the chromakey code and do the color knockout yourself beforehand.
It shouldn't be that hard to rewrite the Chromakey sample app to use a video file as input instead of a camera feed, and it's quite easy to disable the chormakey code.
You'd need to modify the setup on the video input to expect RGBA data instead of RGB or Y/UV. The sample app is set up to use RGB, but I've seen other example apps from Apple that use Y/UV instead.
Have a look at the free "APNG" app on the app store. It shows how an animated PNG (.apng) can be rendered directly to an iOS view. The key is that APNG supports an alpha channel in the file format, so you don't need to mess around with chroma tricks that will not really work for all your video content. This approach is more efficient that multiple layers or chroma tricks since another round of processing is not needed each time a texture is displayed in a loop.
If you want to have a look at a small example xcode project that displays an alpha channel animation on the side of a spinning cube with OpenGL ES2, it can be found at Load OpenGL textures with alpha channel on iOS. The example code shows a simple call to glTexImage2D() that uploads a texture to the graphics card once for each display link callback.