I have an occasional issue with my MTKView renderer stalling on obtaining a currentRenderPassDescriptor for 1.0s. According to the docs, this is either due the view's device not being set (it is) or there are no drawables available.
If there are no drawables available, I don't see a means of just immediately bailing or skipping that video frame. The render loop will stall for 1.0s.
Is there a workaround for this?. Any help would be appreciated.
My workflow is a bunch of kernel shader work then one final vertex shader. I could do the drawing of the final shader onto my own texture (instead of using the currentPassDescriptor), then hoodwink that texture into the view's currentDrawable -- but in the obtaining of that drawable we're back to the same stalling situation.
Should I get rid of MTKView entirely and fall back to using a CAMetalLayer instead? Again, I suspect the same stalling issues will arise. Is there a way to set the maximumDrawableCount on an MTKView like there is on CAMetalLayer?
I'm a little baffled as, according the Metal System Trace, my work is invariably completed under 5.0ms per frame on an iMac 2015 R9 M395.
Related
When i run my app in ios 10 using xcode 8 i am getting following message in debug console, and by UI getting freezed can any one know why this is happening
ERROR
/BuildRoot/Library/Caches/com.apple.xbs/Sources/VectorKit/VectorKit-1228.30.7.17.9/GeoGL/GeoGL/GLCoreContext.cpp
1763: InfoLog SolidRibbonShader: ERROR
/BuildRoot/Library/Caches/com.apple.xbs/Sources/VectorKit/VectorKit-1228.30.7.17.9/GeoGL/GeoGL/GLCoreContext.cpp
1764: WARNING: Output of vertex shader 'v_gradient' not read by
fragment shader
Answer
One of the situations where you might get this warning in Xcode is when using an app that uses shaders such as the Maps app with an MKMapView. You'll find that the map view works as expected without that warning on a real device with real hardware/native OS.
In the sim the SolidRibbonShader fragment shader is not able to read the output of the v_gradient vertex shader probably because it's in beta or there might be an incompatibility between Xcode version and SIM version. However the shaders are recognized on a real device.
Explanation
Those shaders belong to the OpenGL Rendering Pipeline. The Rendering Pipeline is the sequence of steps that OpenGL takes when rendering objects.
The rendering pipeline is responsible for things like applying texture, converting the vertices to the right coordinate system and displaying the character on the screen etc.
There are six stages in this pipeline.
Per-Vertex Operation
Primitive Assembly
Primitive Processing
Rasterization
Fragment Processing
Per-Fragment Operation
Finally, an image appears on the screen of your device. These six stages are called the OpenGL Rendering Pipeline and all data used for rendering must go through it.
What is a shader?
A shader is a small program developed by you that lives in the GPU. A shader is written in a special graphics language called OpenGL Shading Language(GLSL).
A shader takes the place of two important stages in the OpenGL Rendering Pipeline: Per-Vertex Processing and Per-Fragment Processing stage. There is one shader for each of these two stages.
The ultimate goal of the Vertex Shader is to provide the final transformation of the mesh vertices to the rendering pipeline. The goal of the Fragment shader is to provide Coloring and Texture data to each pixel heading to the framebuffer.
Vertex shaders transform the vertices of a triangle from a local model coordinate system to the screen position. Fragment shaders compute the color of a pixel within a triangle rasterized on screen.
Separate Shader Objects Speed Compilation and Linking
Many OpenGL ES apps use several vertex and fragment shaders, and it is often useful to reuse the same fragment shader with different vertex shaders or vice versa. Because the core OpenGL ES specification requires a vertex and fragment shader to be linked together in a single shader program, mixing and matching shaders results in a large number of programs, increasing the total shader compile and link time when you initialize your app.
Update: the issue seems to be gone now on Xcode9/iOS11.
Firstly, the freezing problem happens only when run from Xcode 8 and only on iOS 10 (currently 10.0.2), whether in debug or release mode. MKMapView though seems fine when the app is distributed via App Store or 3rd party ad hoc distribution systems. The warnings you are seeing may or may not be related to the problem, I don't know.
What I've found is that the offending code is in MKMapView's destructor, and it doesn't matter what you do with the map view object or how you configure it, i.e. merely calling
[MKMapView new];
anywhere in your code will freeze the app. The main thread hangs on a semaphore and it's not clear why.
One of the things I've tried was to destroy the map view object in a separate thread but that didn't help. Eventually I decided to retain my map objects at least in DEBUG builds.
NOTE: this is a really sh*tty workaround but at least it will help you to debug your app without freezing. Retaining these objects means your memory usage will grow by about 45-50MB every time you create a view controller with a map.
So, let's say if you have a property mapView, then you can do this in your view controller's dealloc:
- (void)dealloc
{
#if DEBUG
// Xcode8/iOS10 MKMapView bug workaround
static NSMutableArray* unusedObjects;
if (!unusedObjects)
unusedObjects = [NSMutableArray new];
[unusedObjects addObject:_mapView];
#endif
}
To avoid writing to a constant buffer from both the gpu and cpu at the same time, Apple recommends using a triple-buffered system with the help of a semaphore to prevent the cpu getting too far ahead of the gpu (this is fine and covered in at least three Metal videos now at this stage).
However, when the constant resource is an MTLTexture and the AVCaptureVideoDataOutput delegate runs separately than the rendering loop (CADisplaylink), how can a similar triple-buffered system (as used in Apple’s sample code MetalVideoCapture) guarantee synchronization? Screen tearing (texture tearing) can be observed if you take the MetalVideoCapture code and simply render a full screen quad and change the preset to AVCaptureSessionPresetHigh (at the moment the tearing is obscured by the rotating quad and low quality preset).
I realize that the rendering loop and the captureOutput delegate method (in this case) are both on the main thread and that the semaphore (in the rendering loop) keeps the _constantDataBufferIndex integer in check (which indexes into the MTLTexture for creation and encoding), but screen tearing can still be observed, which is puzzling to me (it would make sense if the gpu writing of the texture is not the next frame after encoding but 2 or 3 frames after, but I don’t believe this to be the case). Also, just a minor point: shouldn’t the rendering loop and the captureOutput have the same frame rate for a buffered texture system so old frames aren’t rendered interleaved with recent ones.
Any thoughts or clarification on this matter would be greatly appreciated; there is another example from McZonk, which doesn’t use the triple-buffered system, but I also observed tearing with this approach (but less so). Obviously, no tearing is observed if I use waitUntilCompleted (equivalent to Open GL’s glfinish), but thats like playing an accordion with one arm tied behind your back!
At the moment I am using snapshot to do my picking. I change the render code to render out object ids, grab the snapshot, then take the value for the pixel under the user tap. I think this is quite inefficient though - and I'm getting reports of slowness on some ipads (my mini is fine).
Is it possible to render to the backbuffer, and use a call of glreadpixels to retrieve only the pixel under the user tap without the object-ids being rendered to the screen? I am using GLKView for my rendering. I've tried glreadpixels with my current code - and it always seems to return black. I know that the documentation for GLKView recommends only to use snapshot, but surely it is more efficient for picking to only retrieve a single pixel.
You are correct, a much better way is to render the object ids to the back buffer and read back a particular pixel (or block of pixels).
(If you're doing a lot of selection, you could even use a second offscreen renderbuffer and generate the object ids every frame in a single render pass.)
But you will have to write your own view code to allocate offscreen render buffers, depth buffers, and whatnot. GLKView is a convenience class, a high level wrapper, and the Apple doco specifically says not to mess with the underlying implementation.
Setting up your own GL render buffers isn't too difficult, and there's example code all over the place. I've used the example code on the Apple dev site and from the OpenGL SuperBible.
Actually it is quite possible to read from the backbuffer, even using GLKView. The documentation states that it is not advised - but after a bit of fiddling I got it to work. The only thing which was a problem is that glreadpixels can only take GL_RGBA as argument (not GL_RGB). So long as you ensure that glClear is called after the picking you will not get object ids rendered to the screen.
Using snapshot to do the picking on an ipad mini slowed down the app 50%. Using glReadPixels leads to no noticable slowdown at all. You could do this by allocating an extra framebuffer - but I don't think it is necessary.
To clarify, I know that a texture atlas improves performance when using multiple distinct images. But I'm interested in how things are done when you are not doing this.
I tried doing some frame-by-frame animation manually in custom OpenGL where each frame I bind a new texture and draw it on the same point sprite. It works, but it is very slow compared to the UIImageView ability to abstract the same. I load all the textures up front, but the rebinding is done each frame. By comparison, UIImageView accepts the individual images, not a texture atlas, so I'd imagine it is doing similarly.
These are 76 images loaded individually, not as a texture atlas, and each is about 200px square. In OpenGL, I suspect the bottleneck is the requirement to rebind a texture at every frame. But how is UIImageView doing this as I'd expect a similar bottleneck?? Is UIImageView somehow creating an atlas behind the scenes so no rebinding of textures is necessary? Since UIKit ultimately has OpenGL running beneath it, I'm curious how this must be working.
If there is a more efficient means to animate multiple textures, rather than swapping out different bound textures each frame in OpenGL, I'd like to know, as it might hint at what Apple is doing in their framework.
If I did in fact get a new frame for each of 60 frames in a second, then it would take about 1.25 seconds to animate through my 76 frames. Indeed I get that with UIImageView, but the OpenGL is taking about 3 - 4 seconds.
I would say your bottleneck is somewhere else. The openGL is more then capable doing an animation the way you are doing. Since all the textures are loaded and you just bind another one each frame there is no loading time or anything else. Consider for a comparison I have an application that can in runtime generate or delete textures and can at some point have a great amount of textures loaded on the GPU, I have to bind all those textures every frame (not 1 every frame), using all from depth buffer, stencil, multiple FBOs, heavy user input, about 5 threads bottlenecked into 1 to process all the GL code and I have no trouble with the FPS at all.
Since you are working with the iOS I suggest you run some profilers to see what code is responsible for the overhead. And if for some reason your time profiler will tell you that the line with glBindTexture is taking too long I would still say that the problem is somewhere else.
So to answer your question, it is normal and great that UIImageView does its work so smoothly and there should be no problem achieving same performance with openGL. THOUGH, there are a few things to consider at this point. How can you say that image view does not skip images, you might be setting a pointer to a different image 60 times per second but the image view might just ask itself 30 times per second to redraw and when it does just uses a current image assigned to it. On the other hand with your GL code you are forcing the application to do the redraw 60FPS regardless to if it is capable of doing so.
Taking all into consideration, there is a thing called display link that apple developers created for you. I believe it is meant for exactly what you want to do. The display link will tell you how much time has elapsed between frames and by that key you should ask yourself what texture to bind rather then trying to force them all in a time frame that might be too short.
And another thing, I have seen that if you try to present render buffer at 100 FPS on most iOS devices (might be all), you will only get 60 FPS as the method to present render buffer will pause your thread if it has been called in less then 1/60s. That being said it is rather impossible do display anything at all at 60 FPS on iOS devices and everything running 30+ FPS is considered good.
"not as a texture atlas" is the sentence that is a red flag for me.
USing a texture atlas is a good thing....the texture is loaded into memory once and then you just move the rectangle position to play the animation. It's fast because its already all in memory. Any operation which involves constantly loading and reloading new image frames is going to be slower than that.
You'd have to post source code to get any more exact an answer than that.
I'm developing an iPad app that uses large textures in OpenGL ES. When the scene first loads I get a large black artifact on the ceiling for a few frames, as seen in the picture below. It's as if higher levels of the mipmap have not yet been filled in. On subsequent frames, the ceiling displays correctly.
This problem only began showing up when I started using mipmapping. One possible explanation is that the glGenerateMipmap() call does its work asynchronously, spawning some mipmap creation worker (in a separate process, or perhaps in the GPU) and returning.
Is this possible, or am I barking up the wrong tree?
Within a single context, all operations will appear to execute strictly in order. However, in your most recent reply, you mentioned using a second thread. To do that, you must have created a second shared context: it is always illegal to re-enter an OpenGL context. If already using a shared context, there are still some synchronization rules you must follow, documented at http://developer.apple.com/library/ios/ipad/#DOCUMENTATION/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/WorkingwithOpenGLESContexts/WorkingwithOpenGLESContexts.html
It should be synchronous; OpenGL does not in itself have any real concept of threading (excepting the implicit asynchronous dialogue between CPU and GPU).
A good way to diagnose would be to switch to GL_LINEAR_MIPMAP_LINEAR. If it's genuinely a problem with lower resolution mip maps not arriving until later then you'll see the troublesome areas on the ceiling blend into one another rather than the current black-or-correct effect.
A second guess, based on the output, would be some sort of depth buffer clearing issue.
I followed #Tommy's suggestion and switched to GL_LINEAR_MIPMAP_LINEAR. Now the black-or-correct effect changed to a fade between correct and black.
I guess that although we all know that OpenGL is a pipeline (and therefore asynchronous unless you are retrieving state or explicity synchronizing), we tend to forget it. I certainly did in this case, where I was not drawing, but loading and setting up textures.
Once I confirmed the nature of the problem, I added a glFinish() after loading all my textures, and the problem went away. (Btw, my draw loop is in the foreground and my texture loading loop - because it is so time consuming and would impair interactivity - is in the background. Also, since this may vary between platforms, I'm using iOS5 on an iPad 2)