I have a complex pre-rendered scene which I would like to use as a backdrop in a 3D iPad game which uses a static camera.
For each frame redraw the screen will be erased to this background. The part I do not know how to do is set the depth buffer to the one stored in this pre-rendered image, so that dynamic 3D objects will respect the depth information in said image.
Is there any way to achieve this on an iPad, using opengl es 2.0?
I looked into several approaches, but could not find anything suitable so far.
Related
I'm working on a university project for which I want to texturize meshes which were generated using the new LiDAR sensor on recent Apple devices at run-time.
I've read some research papers about texturing 3D reconstructions and I want to try a 'simple' approach were mesh faces (triangles) are transformed to image-space of key-frames that were taken with the camera of the device. Then, I want to perform a pixel-lookup at the position of the triangle in the image and then add that part of the image to a texture map. When I keep track of all triangles in the mesh and determine its corresponding color in a relevant key-frame, I should be able to infer the texture / color of all triangles in the mesh.
Here is a demonstration of the aforementioned principle:
I need to do this natively in swift and therefore I will be using Metal for this. However, I have not worked with Metal before and my question is whether it is possible to "stich together" a texture from different image sources dynamically at run-time?
I'm working on an Open GL app that uses 1 particularly large texture 2250x1000. Unfortunately, Open GL ES 2.0 doesn't support textures larger than 2048x2048. When I try to draw my texture, it appears black. I need a way to load and draw the texture in 2 segments (left, right). I've seen a few questions that touch on libpng, but I really just need a straight forward solution for drawing large textures in opengl es.
First of all the texture size support depends on device, I believe iPad 3 supports 4096x4096 but don't mind that. There is no way to push all those data as they are to most devices onto 1 texture. First you should ask yourself if you really need such a large texture, will it really make a difference if you resample it down to 2048x_. If the answer is NO you will need to break it at some point. You could cut it by half in width and append of the cut parts to the bottom of the texture resulting in 1125x2000 texture or simply create 2 or more textures and push to them certain parts of the texture image. In any of the cases you might have trouble with texture coordinates but this all heavily depends on what you are trying to do, what is on that texture (a single image or parts of a sophisticated model; color mapping or some data you can not interpolate; do you create it at load time or it is modified as it goes...). Maybe some more info could help us solve your situation more specifically.
I saw that someone has made an app that tracks your feet using the camera, so that you can kick a virtual football on your iPhone screen.
How could you do something like this? Does anyone know of any code examples or other information about using the iPhone camera for detecting objects and tracking them?
I just gave a talk at SecondConf where I demonstrated the use of the iPhone's camera to track a colored object using OpenGL ES 2.0 shaders. The post accompanying that talk, including my slides and sample code for all demos can be found here.
The sample application I wrote, whose code can be downloaded from here, is based on an example produced by Apple for demonstrating Core Image at WWDC 2007. That example is described in Chapter 27 of the GPU Gems 3 book.
The basic idea is that you can use custom GLSL shaders to process images from the iPhone camera in realtime, determining which pixels match a target color within a given threshold. Those pixels then have their normalized X,Y coordinates embedded in their red and green color components, while all other pixels are marked as black. The color of the whole frame is then averaged to obtain the centroid of the colored object, which you can track as it moves across the view of the camera.
While this doesn't address the case of tracking a more complex object like a foot, shaders like this should be able to be written that could pick out such a moving object.
As an update to the above, in the two years since I wrote this I've now developed an open source framework that encapsulates OpenGL ES 2.0 shader processing of images and video. One of the recent additions to that is a GPUImageMotionDetector class that processes a scene and detects any kind of motion within it. It will give you back the centroid and intensity of the overall motion it detects as part of a simple callback block. Using this framework to do this should be a lot easier than rolling your own solution.
I'm using a texture cache to draw video frames to the screen, just like the RosyWriter sample application from Apple.
I want to downsample an image from 1080p down to around 320x480 (for various reasons, I don't want to capture at a lower resolution) and use mipmap filtering to get rid of aliasing. However, when I try adding:
glGenerateMipmap(CVOpenGLESTextureGetTarget(inputTexture));
glTexParameteri(CVOpenGLESTextureGetTarget(inputTexture), GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
I just get a black screen, as though the mipmaps aren't being generated. I'm rendering offscreen from one texture to another. Both source and destination are mapped to pixel buffers using texture caches.
Mipmaps can only be generated for power-of-two sized textures. None of the video frame sizes returned by the iOS cameras that I can think of have power-of-two dimensions. For using the texture caches while still generating mipmaps, I think you'd have to do something like do an offscreen re-render of the texture to a power-of-two FBO backed by a texture, then generate a mipmap for that.
That said, this is probably not the best way to accomplish what you want. Mipmaps only help when making a texture smaller on the screen, not making it larger. Also, they are pretty slow to generate at runtime, so this would drag your entire video processing down.
What kind of aliasing are you seeing when you zoom in? The normal hardware texture filtering should produce a reasonably smooth image when zoomed in on a video frame. As an example of this, grab and run the FilterShowcase sample from my GPUImage framework and look at the Crop filter. Zooming in on a section of the video that way seems to smooth things out pretty nicely, just using hardware filtering.
I do employ mipmaps for smooth downsampling of large images in the framework (see the GPUImagePicture when smoothlyScaleOutput is set to YES), but again that's for shrinking an image, not zooming in on it.
I know it is possible, and a lot faster than using GDI+. However I haven't found any good example of using DirectX to resize an image and save it to disk. I have implemented this over and over in GDI+, thats not difficult. However GDI+ does not use any hardware acceleration, and I was hoping to get better performance by tapping into the graphics card.
You can load the image as a texture, texture-map it onto a quad and draw that quad in any size on the screen. That will do the scaling. Afterwards you can grab the pixel-data from the screen, store it in a file or process it further.
It's easy. The basic texturing DirectX examples that come with the SDK can be adjusted to do just this.
However, it is slow. Not the rendering itself, but the transfer of pixel data from the screen to a memory buffer.
Imho it would be much simpler and faster to just write a little code that resizes an image using bilinear scaling from one buffer to another.
Do you really need to use DirectX? GDI+ does the job well for resizing images. In DirectX, you don't really need to resize images, as most likely you'll be displaying your images as textures. Since textures can only applies on 3d object (triangles/polygons/mesh), the size of the 3d object and view port determines the actual image size displayed. If you need to scale your texture within the 3d object, just play the texture coordinate or matrix.
To manipute the texture, you can use alpha blending, masking and all sort of texture manipulation technique, if that's what you're looking for. To manipulate individual pixel like GDI+, I still think GDI+ is the way to do. DirectX was never mend to do image manipulation.