I am trying to play a video using OpenGL ES 2.0 in iOS. I am not able to get a sample code or starting point of how to achieve this. Can anybody help me with this?
What you are looking for is getting a raw buffer for the video in real time. I believe you need to look into AVFoundation and somehow extract the CVPixelBufferRef. If I remember correctly you have a few ways; one is on demand at specific time, another for processing where you will get a fast iteration of the frames in a block, and the one you probably need is to receive the frames in real time. So with this you can extract a raw RGB buffer which needs to be pushed to the texture and then drawn to the render buffer.
I suggest you create a texture once (per video) and try making it as small as possible but ensure that the video frame will fit. You might need the POT (power of two) textures so to get the texture dimension from video width you need something like:
GLInt textureWidth = 1.0f;
while(textureWidth<videoWidth) textureWidth <<= 1; // Multiplies by 2
So the texture size is expected to be larger then the video. To push the data to the texture you then need to use texture subimage glTexSubImage2D. Which expects a pointer to your raw data and rectangle parameters where to save the data which are then (0, 0, sampleWidth, sampleHeight). Also then the texture coordinates must computed so they are not in range [0, 1] but rather for x: [0, sampleWidth/textureWidth].
So then you just need to put it all together:
Have a system to keep generating the video raw sample buffers
Generate a texture to fit video size
On new sample update the texture using glTexSubImage2D (watch out for threads)
After the data is loaded into the texture draw the texture as full screen rectangle (watch out for threads)
You might need to watch out for video orientation, transformation. So if possible do test your system with a few videos that have been recorded on the device in different orientations. I think there is now a support to receive the buffers already correctly oriented. But by default the sample at least used to be "wrong"; the portrait recorded video still had the samples in landscape but a transformation matrix or orientation was given with the asset.
Related
I would like to generate a grid picture or bitmap or anything similar with raw pixel data in swift. Since the pixel location, image size etc. are not determined before the user opens the app or presses a refresh button I need a fast way to generate 2732x2048 or more individual pixels and display them on the screen.
First I did use UIGraphicsBeginImageContextWithOptions and drew each pixel with a 1x1 CGRect but this obviously did not scale well.
Afterwards I have used this approach: Pixel Array to UIImage in Swift
But this is still kind of slow with the bigger screens.
Could something like this be done with MetalKit? I would assume that a lower api does render something like this way faster?
Or is there any better way to process something like this in-between MetalKit and CoreGraphics?
Some info regarding the structure of my data:
There is a struct with the pixel color data red, green, blue, alpha for each individual pixel stored as an Array and two image size variables: imageHeight and imageWidth.
The most performant way to do that is to use Metal Compute Function.
Apple has a good documentation to illustrate GPU programming.
Performing Calculations on a GPU
Processing a Texture in a Compute Function
I need to grab the screen pixels into a texture to perform post processing.
Previously, i have been using BlitCommandEncoder to copy from texture to texture. Source texture being the MTLDrawable texture, onto my destination texture. They both have the same MTLPixelFormatBGRA8Unorm so everything works just fine.
However, now i need to use a frame buffer color attachment texture of MTLPixelFormatRGBA16Float for HDR rendering. So, when i am grabbing the screen pixels, i am actually grabbing from this color attachment texture instead of the Drawable texture. And i am getting this error:
[MTLDebugBlitCommandEncoder internalValidateCopyFromTexture:sourceSlice:sourceLevel:sourceOrigin:sourceSize:toTexture:destinationSlice:destinationLevel:destinationOrigin:options:]:447: failed assertion [sourceTexture pixelFormat](MTLPixelFormatRGBA16Float) must equal [destinationTexture pixelFormat](MTLPixelFormatBGRA8Unorm)
I don't think i need to change my destination texture to RGBA16Float format? Because that will take up double the memory. One full screen texture (color attachment) with that format should be enough for HDR to work right?
Is there other method to successfully perform this kind of copy?
On openGL there is no error when copying with glCopyTexImage2D
Metal automatically converts from source to destination format during rendering. So you could just do a no-op rendering pass to perform the conversion.
Alternatively, if you want to avoid boilerplate no-op rendering code, you can use the MPSImageConversion performance shader that's basically doing the same.
ARKit runs at 60 frames/sec, which equates to 16.6ms per frame.
My current code to convert the CVPixelBufferRef (kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format) to a cv::Mat (YCrCb) runs in 30ms, which causes ARKit to stall and everything to lag.
Does anyone have any ideas on how to to a quicker conversion or do I need to drop the frame rate?
There is a suggestion by Apple to use Metal, but I'm not sure how to do that.
Also I could just take the grayscale plane, which is the first channel, which runs in <1ms, but ideally I need the colour information as well.
In order to process an image in a pixel buffer using Metal, you need to do following.
Call CVMetalTextureCacheCreateTextureFromImage to create CVMetalTexture object on top of the pixel buffer.
Call CVMetalTextureGetTexture to create a MTLTexture object, which Metal code (GPU) can read and write.
Write some Metal code to convert the color format.
I have an open source project (https://github.com/snakajima/vs-metal), which processes pixel buffers (from camera, not ARKit) using Metal. Feel free to copy any code from this project.
I tried to convert Ycbcr to RGB, do image processing in RGB mat and convert it back to Ycbcr, it worked very slowly. I suggest only do that with a static image. For realtime processing, we should process directly in cv::Mat. ARFrame.capturedImage is Ycbcr buffer. So, the solution is
Sperate the buffer to 2 cv::Mat (yPlane and cbcrPlane). Keep in mind, we do not clone memory, we create 2 cv::Mat with base addresses is yPlane address and cbcrPlane address.
Do image process on yPlane and cbcrPlane, size(cbcrPlane) = size(yPlane) / 2.
You can check out my code here: https://gist.github.com/ttruongatl/bb6c69659c48bac67826be7368560216
I am trying to write a little script to apply texture to rectangular cuboids. To accomplish this, I run through the scenegraph, and wherever I find the SoIndexedFaceSet Nodes, I insert a SoTexture2 Node before that. I put my image file in the SoTexture2 Node. The problem I am facing is that the texture is applied correctly to 2 of the faces(say face1 and face2), in the Y-Z plane, but for the other 4 planes, it just stretches the texture at the boundaries of the two faces(1 and 2).
It looks something like this.
The front is how it should look, but as you can see, on the other two faces, it just extrapolates the corner values of the front face. Any ideas why this is happening and any way to avoid this?
Yep, assuming that you did not specify texture coordinates for your SoIndexedFaceSet, that is exactly the expected behavior.
If Open Inventor sees that you have applied a texture image to a geometry and did not specify texture coordinates, it will automatically compute some texture coordinates. Of course it's not possible to guess how you wanted the texture to be applied. So it computes the bounding box then computes texture coordinates that stretch the texture across the largest extent of the geometry (XY, YZ or XZ). If the geometry is a cuboid you can see the effect clearly as in your image. This behavior can be useful, especially as a quick approximation.
What you need to make this work the way you want, is to explicitly assign texture coordinates to the geometry such that the texture is mapped separately to each face. In Open Inventor you can actually still share the vertices between faces because you are allowed to specify different vertex indices and texture coordinate indices (of course this is only more convenient for the application because OpenGL doesn't support this and Open Inventor has to re-shuffle the data internally). If you applied the same texture to an SoCube node you would see that the texture is mapped separately to each face as expected. That's because SoCube defines texture coordinates for each face.
I'm currently creating an iOS app and I'm having trouble understanding the relationship between taking pixels from an ofGrabber and drawing them using an ofTexture.
My current code:
In setup():
//Set iOS orientation
ofSetOrientation(OF_ORIENTATION_90_LEFT);
//Inits the camera to specified dimensions and sets up texture to display on screen
grabber.initGrabber(640, 480, OF_PIXELS_BGRA); //Options: 1280x720, 640x480
//Allocate opengl texture
tex.allocate(grabber.width, grabber.height, GL_RGB);
//Create pix array large enough to store an rgb value for each pixel
//pix is a global that I use to do pixel manipulation before drawing
pix = new unsigned char[grabber.width * grabber.height * 3];
In update()
//Loads the new pixels into the opengl texture
tex.loadData(pix, grabber.width, grabber.height, GL_RGB);
In draw():
CGRect screenBounds = [[UIScreen mainScreen] bounds];
CGSizeMake screenSize = CGSizeMake(screenBounds.size.width, screenBounds.size.height);
//Draws the texture we generated onto the screen
//On 1st generation iPad mini: width = 1024, height = 768
tex.draw(0, 0, screenSize.height, screenSize.width); //Reversed for 90 degree rotation
What I'm wondering:
1) Why does the ofGrabber and the ofTexture use seemingly different pixel formats? (These formats are the same used in the VideoGrabberExample)
2) What exactly is the texture drawing with the resolution? I'm loading the pix array into the texture. The pix array represents a 640x480 image, while the ofTexture is drawing a 1024x768 (768x1024 when rotated) image to the screen. How is it doing this? Does it just scale everything up since the aspect ratio is basically the same?
3) Is there a list anywhere that describes the OpenGL and OpenFrameworks pixel formats? I've searched for this but haven't found much. For example, why is it OF_PIXELS_BGRA instead of OF_PIXELS_RGBA? For that matter, why does my code even work if I'm capturing BGRA formatted data (which I assume included a gamma value) yet I am only drawing RGB (and you can see that my pix array is sized for RGB data).
I might also mention that in main() I have:
ofSetupOpenGL(screenSize.height, screenSize.width, OF_FULLSCREEN);
However, changing the width/height values above seem to have no effect whatsoever on my app.
ofGrabber is CPU based, so it uses OF_PIXELS_BGRA by the programmers choice. It is common for some cameras to have BGRA pixel order, so this just avoids the grabber to perform a costly memcpy when grabbing from the source. ofTexture maps GPU memory, so it maps to what you'll see on screen (RGB). Note that GL_RGB is an OpenGL definition.
ofTexture does scale to whatever you tell it to. This is done in GPU so it's quite cheap. It does not need to have the same aspect ratio.
This is quite up to the programmer or your requirements. Some cameras provide BGRA streams, other cameras or files provide RGB directly, or even YUV-I420. Color formats are very heterogeneous. OpenFrameworks will handle conversions in most cases, look into ofPixels to see where and how it's used. In a nutshell:
OF_PIXELS_XXX : Used by ofPixels, basically a RAM mapped bitmap
OF_IMAGE_XXX : Used by ofImage, which wrapps ofPixels and makes it simpler to use
GL_XXX : Used by OpenGL and ofTexture, low level GPU Memory mapped