I'm currently developping an iPad app which is using OpenGL to draw some very simple (no more than 1000 or 2000 vertices) rotating models in multiple OpenGL views.
There are currently 6 view in a grid, each one running its own display link to update the drawing. Due to the simplicity of the models, it's by far the simplest method to do it, I don't have the time to code a full OpenGL interface.
Currently, it's doing well performance-wise, but there are some annoying glitches. The first 3 OpenGL views display without problems, and the last 3 only display a few triangles (while still retaining the ability to rotate the model). Also there are some cases where the glDrawArrays call is going straight into EXC_BAD_ACCESS (especially on the simulator), which tell me there is something wrong with the buffers.
What I checked (as well as double- and triple-checked) is :
Buffer allocation seems OK
All resources are freed on dealloc
The instruments show some warnings, but nothing that seems related
I'm thinking it's probably related to my having multiple views drawing at the same time, so is there any known thing I should have done there? Each view has its own context, but perhaps I'm doing something wrong with that...
Also, I just noticed that in the simulator, the afflicted views are flickering between the right drawing with all the vertices and the wrong drawing with only a few.
Anyway, if you have any ideas, thanks for sharing!
Okay, I'm going to answer my own question since I finally found what was going on. It was a small missing line that was causing all those problems.
Basically, to have multiple OpenGL views displayed at the same time, you need :
Either, the same context for every view. Here, you have to take care not to draw with multiple threads at the same time (i.e. lock the context somehow, as explained on this answer. And you have to re-bind the frame- and render-buffers every time on each frame.
Or, you can use different contexts for each view. Then, you have to re-set the context on each frame, because other display links, could (and would, as in my case) cause your OpenGL calls to use the wrong data. Also, there is no need for re-binding frame- and render-buffers since your context is preserved.
Also, call glFlush() after each frame, to tell the GPU to finish rendering each frame fully.
In my case (the second one), the code for rendering each frame (in iOS) looks like :
- (void) drawFrame:(CADisplayLink*)displayLink {
// Set current context, assuming _context
// is the class ivar for the OpenGL Context
[EAGLContext setCurrentContext:_context]
// Clear whatever you want
glClear (GL_DEPTH_BUFFER_BIT | GL_COLOR_BUFFER_BIT);
// Do matrix stuff
...
glUniformMatrix4fv (...);
// Set your viewport
glViewport (0, 0, self.frame.size.width, self.frame.size.height);
// Bind object buffers
glBindBuffer (GL_ARRAY_BUFFER, _vertexBuffer);
glVertexAttribPointer (_glVertexPositionSlot, 3, ...);
// Draw elements
glDrawArrays (GL_TRIANGLES, 0, _currentVertexCount);
// Discard unneeded depth buffer
const GLenum discard[] = {GL_DEPTH_ATTACHMENT};
glDiscardFramebufferEXT (GL_FRAMEBUFFER, 1, discard);
// Present render buffer
[_context presentRenderbuffer:GL_RENDERBUFFER];
// Unbind and flush
glBindBuffer (GL_ARRAY_BUFFER, 0);
glFlush();
}
EDIT
I'm going to edit this answer, since I found out that running multiple CADisplayLinks could cause some issues. You have to make sure to set the frameInterval property of your CADisplayLink instance to something other than 0 or 1. Else, the run loop will only have time to call the first render method, and then it'll call it again, and again. In my case, that was why only one object was moving. Now, it's set to 3 or 4 frames, and the run loop has time to call all the render methods.
This applies only to the application running on the device. The simulator, being very fast, doesn't care about such things.
It gets tricky when you want multiple UIViews that are openGLViews,
on this site you should be able to read all about it: Using multiple openGL Views and uikit
Related
I'm on Yosemite 10.10.5 and Xcode 7, using Swift to make a game targeting iOS 8 and above.
EDIT: More details that might be useful: This is a 2D puzzle/arcade game where the player moves stones around to match them up. There is no 3D rendering at all. Drawing is already too slow and I haven't even gotten to explosions with debris yet. There is also a level fade-in, very concerning. But this is all on the simulator so far. I don't yet have an actual iPhone to test with yet and I'm betting the actual device will be at least a little faster.
I have my own Draw2D class, which is a type of UIView, set up as in this tutorial. I have a single NSTimer which initiates the following chain of calls in Draw2D:
[setNeedsDisplay]; // which calls drawRect, which is the master draw function of Draw2D
drawRect(rect: CGRect)
{
scr_step(); // the master update function, which loops thru all objects and calls their individual update functions. I put it here so that updating and drawing are always in sync
CNT = UIGraphicsGetCurrentContext(); // get the curret drawing context
switch (Realm) // based on what realm im in, call the draw function for that realm
{
case rlm.intro: scr_draw_intro();
case rlm.mm: scr_draw_mm();
case rlm.level: scr_draw_level(); // this in particular loops thru all objects and calls their individual draw functions
default: return;
}
var i = AARR.count - 1; // loop thru my own animation objects and draw them too, note it's iterating backwards because sometimes they destroy themselves
while (i >= 0)
{
let A = AARR[i];
A.scr_draw();
i -= 1;
}
}
And all the drawing works fine, but slow.
The problem is now I want to optimize drawing. I want to draw only in the dirty rectangles that need drawing, not the whole screen, which is what setNeedsDisplay is doing.
I could not find any tutorials or good example code for this. The closest I found was apple's documentation here, but it does not explain, among other things, how to get a list of all dirty rectangles so far. It does not also explicitly state if the list of dirty rectangles is automatically cleared at the end of each call to drawRect?
It also does not explain if I have to manually clip all drawing based on the rectangles. I found conflicting info about that around the web, apparently different iOS versions do it differently. In particular, if I'm gonna hafta manually clip things then I don't see the point of apple's core function in the first place. I could just maintain my own list of rectangles and manually compare each drawing destination rectangle to the dirty rectangle to see if I should draw anything. That would be a huge pain, however, because I have a background picture in each level and I would hafta draw a piece of it behind every moving object. What I'm really hoping for is the proper way to use setNeedsDisplayInRect to let the core framework do automatic clipping for everything that gets drawn on the next draw cycle, so that it automatically draws only that piece of the background plus the moving object on top.
So I tried some experiments: First in my array of stones:
func scr_draw_stone()
{
// the following 3 lines are new, I added them to try to draw in only dirty rectangles
if (xvp != xv || yvp != yv) // if the stone's coordinates have changed from its previous coordinates
{
MyD.setNeedsDisplayInRect(CGRectMake(x, y, MyD.swc, MyD.shc)); // MyD.swc is Draw2D's current square width in points, maintained to softcode things for different screen sizes.
}
MyD.img_stone?.drawInRect(CGRectMake(x, y, MyD.swc, MyD.shc)); // draw the plain stone
img?.drawInRect(CGRectMake(x, y, MyD.swc, MyD.shc)); // draw the stone's icon
}
This did not seem to change anything. Things were drawing just as slow as before. So then I put it in brackets:
[MyD.setNeedsDisplayInRect(CGRectMake(x, y, MyD.swc, MyD.shc))];
I have no idea what the brackets do, but my original setNeedsDisplay was in brackets just like they said to do in the tutorial. So I tried it in my stone object, but it had no effect either.
So what do I need to do to make setNeedsDisplayInRect work properly?
Right now, I suspect there's some conditional check I need in my master draw function, something like:
if (ListOfDirtyRectangles.count == 0)
{
[setNeedsDisplay]; // just redraw the whole view
}
else
{
[setNeedsDisplayInRect(ListOfDirtyRecangles)];
}
However I don't know the name of the built-in list of dirty rectangles. I found this saying the method name is getRectsBeingDrawn, but that is for Mac OSX. It doesn't exist in iOS.
Can anyone help me out? Am I on the right track with this? I'm still fairly new to Macs and iOS.
You should really avoid overriding drawRect if at all possible. Existing view/technologies take advantage of any hardware capabilities to make things a lot faster than manually drawing in a graphics context could, including buffering the contents of views, using the GPU, etc. This is repeated many times in the "View Programming Guide for iOS".
If you have a background and other objects on top of that, you should probably use separate views or layers for those rather than redraw them.
You may also consider technologies such as SpriteKit, SceneKit, OpenGL ES, etc.
Beyond that, I'm not quite sure I understand your question. When you call setNeedsDisplayInRect, it will add that rect to those that need to be redrawn (possibly merging with rectangles that are already in the list). drawRect: will then be called a bit later to draw those rectangles one at a time.
The whole point of the setNeedsDisplayInRect / drawRect: separation is to make sure multiple requests to redraw a given part of the view are merged together, and drawing only happens once per redraw cycle.
You should not call your scr_step method in drawRect:, as it may be called multiple times in a cycle redraw cycle. This is clearly stated in the "View Programming Guide for iOS" (emphasis mine):
The implementation of your drawRect: method should do exactly one
thing: draw your content. This method is not the place to be updating
your application’s data structures or performing any tasks not related
to drawing. It should configure the drawing environment, draw your
content, and exit as quickly as possible. And if your drawRect: method
might be called frequently, you should do everything you can to
optimize your drawing code and draw as little as possible each time
the method is called.
Regarding clipping, the documentation of drawRect states that:
You should limit any drawing to the rectangle specified in the rect
parameter. In addition, if the opaque property of your view is set to
YES, your drawRect: method must totally fill the specified rectangle
with opaque content.
Not having any idea what your view shows, what the various method you call do, what actually takes time, it's difficult to provide much more insight into what you could do. Provide more details into your actual needs, and we may be able to help.
I am working with some stuff in Core Graphic's and I am looking for some additional clarification regarding a couple of topics.
drawRect:
I have an understanding of this and know it is where all of the drawing aspect's of a UIView goes, but am just unclear as to what is happening behind the scene's. What happen's when I create a UIView and fill out drawRect then set another object's UIView to be that custom view? When is drawRect being called?
CGGraphicsContext:
I know what the purpose of this is and understand the concept, but I can't see exactly how it is working.
For example:
CGContextSaveGState(context);
CGContextAddRect(context, rect);
CGContextClip(context);
CGContextDrawLinearGradient(context, gradient, startPoint, endPoint, 0);
CGContextRestoreGState(context);
The code above is in my app and work's correctly. The thing that confuses me is how it is working. The idea of saving/restoring a context makes sense, but it appears like I am literally saving a context, using that exact same context to make change's, then restoring the same context once again. It just seem's like I am saving a context and then writing on top of that context, only to restore it. How is it getting saved to a point where when you restore it, it is a different instance of the context than what was just used to make changes? You use the same reference of the variable context in every situation.
Lastly I would appreciate any resource's for practice project's or example's on using Core Graphics. I am looking to improve my skill in the matter since I obviously don't have much at the current time.
What happen's when I create a UIView and fill out drawRect then set another object's UIView to be that custom view? When is drawRect being called?
Adding a view to a 'live' view graph marks the view's frame as in need of display. The main run loop then creates and coalesces invalid rects and soon returns to invoke drawing. It does not draw immediately upon invalidation. This is a good thing because resizing, for example, would result in significant overdrawing -- redundant work which would kill many apps' drawing performance. When drawing, a context is created to render to -- which ultimately outputs to its destination.
Graphics Contexts are abstractions which are free to work optimally for their destination -- a destination could be a device/screen, bitmap, PDF, etc.. However, a context handle (CGContextRef) itself refers to a destination and holds a set of parameters regarding its state (these parameters are all documented here). These parameter sets operate like stacks: Push = CGContextSaveGState, Pop = CGContextRestoreGState. Although the context pointer isn't changing, the stack of parameter sets is changing.
As far as resources, see Programming with Quartz. It's 8 years old now, and was originally written for OS X -- but that ultimately doesn't matter a whole lot because the fundamentals of the drawing system and APIs really haven't evolved significantly since then -- And that is what you intend to focus on. The APIs have been extended, so it would be good to review which APIs were introduced since 10.4 and see what problems they solve, but it's secretly a good thing for you because it helps maintain focus on the fundamental operation of the drawing system. Note that some functionalities were excluded from iOS (e.g. often due to floating point performance and memory constraints, I figure), so every example may not be usable on iOS, but I know of no better guide.
Tip: Your drawing code can be easily reused on OS X and iOS if you use Quartz rather than AppKit/UIKit. Plus, the Quartz APIs have a lower update frequency (i.e. the APIs tend to be longer lived).
-drawRect: gets called at some point after you (e.g. your view controller) have called the view's method -setNeedsDisplay or -setNeedsDisplayInRect:.
Saving the graphics state pushes the current graphics state onto a stack. The graphics state contains fill and stroke setting, the current transformation matrix etc. See Apple's documentation for details.
The Quartz 2D Programming Guide doesn't contain many examples but is otherwise quite thorough.
With quartz/ core graphics the context is literally a set of current parameters to use to draw the next drawing command on top of the previous drawing.
Saving the state let's you save all those parameters for later drawing commands that will reuse them.
Then you can set up a different set of parameters for some drawing commands.
Restoring the state gets you back to where you were.
I recommend the book
Programming with Quartz
2D and PDF Graphics in Mac OS X
Though a bit dated in some ways, it will really teach you how quartz / core graphics really flows.
Ok this is a very very deep topic to talk about. I'll explain a few things to my understanding & try to keep it simple. If I'm mistaken I hope someone can correct me out.
first of all there is concept of onscreen drawing and offscreen drawing. On screen drawing is taken place in GPU where offscreen drawing is taken place in CPU to draw things and then its given to GPU to display on the screen. Thats where drawRect() comes in to place (drawrect is only 1 way of doing the offscreen drawings btw). This is why in the drawRect template method (you will see when you make a subclass of UIView) there is a comment by Apple telling
"Only override drawRect: if you perform custom drawing. An empty implementation adversely affects performance during animation"
The reason is whenever there is drawRect method, the iOS would have to ask the CPU to takecare of the drawing which takes place in drawRect and hand it over to the GPU. (Dont get the idea that this is a bad thing :) ). So this is what happens in drawRect in an abstract level.
Now to the question of why save & restore same context over and over. Have you tried to read the description of the method in apple doc about save/restore context ? If you have, you'd notice that it shows all the graphical states which would be affected by this. Ok how does this help ?
Consider something like this. Lets say you're drawing on a rectangle where you have to limit this next part of the drawing on the right half of it and use shadows and antialiasing, etc. You can save your context before drawing the right side and set whatever properties you want and once you finish that, you can simply restore the context and you can continue with all the settings you had before without explicitly setting them again. It's a good practice as well when you do complex drawings as otherwise it would have weird outcomes you might not expect. something like this below
- drawRect()
{
CGContextSaveGState(context);
drawLeftPart(); // - 1
drawRightPart(); // - 2
someOtherDrawing(); // - 3
CGContextRestoreGState(context);
}
- drawLeftPart()
{
CGContextSaveGState(context);
// do your drawing
CGContextRestoreGState(context);
}
- drawRightPart()
{
CGContextSaveGState(context);
// do your drawing
CGContextRestoreGState(context);
}
- someOtherDrawing()
{
CGContextSaveGState(context);
// do your drawing
CGContextRestoreGState(context);
}
Now what ever properties you set in part 1 wont affect drawing of part 2 & 3 so forth.
Hope this helps,
I currently have a free hand drawing iPad app, that adds lines to a mutable path via quad curves in the touches methods then calls setNeedsDisplayInRect on the new area.
Problem is when the drawing (path) gets rather large, it takes longer to redraw, and begins to bog down. As well as whenever the user changes the brush size or color, it applies this to overlapping parts of the previously drawn path on redraw.
To counter this, I call renderInContext in a background thread in touchesEnded, and merge this with another UIImage in an imageview behind the draw view. Then clear the draw view.
This also helps so when the user hits save, the drawing is usually already rendered in a single UIImage - ready to go.
This works fine on other devices, but on he iPad 3 retina display, the performance is really awful and tends to crash whenever the user lifts his finger multiple times when drawing quickly.
I am seeking any type of advice for best practice in handling this type of situation? Aside from adding additional views to render off of in the background to prevent the main and background thread from accessing the same view at a time - which sounds rather hack-ish - I feel like I'm beating a dead horse?
In my current app, I made a working implementation that works fine on iPad 2 as well as 3, regardless of path length or number of paths. It seems that the graphics card is better at drawing lots of small paths then a few large paths, and either one is faster than rendering an image into a context. So, what I do is even if the user is continuously drawing, I break the path into many smaller paths and add those to an array. This approach gives me one advantage, and one disadvantage.
Advantage: The ability to zoom and redraw the image crisply
Disadvantage: Can't do pixel perfect erasing
As far as multiple colors, I made a subclass of UIBezierPath that includes a color property. Since colors are now serializable via NSCoding, they are easily saveable. In addition, I have a "stroke" object, which holds all of the paths the user created in one continuous stroke. This way I can handle undo / redo correctly.
Hope this info helps.
I am running the boiler plate OpenGL example code that XCode creates for an OpenGL project for iOS. This sets up a simple ViewController and uses GLKit to handle the rest of the work.
All the update/draw functionality of the application is in C++. It is cross platform.
There is a lot of framebuffer creation going on. The draw phase renders to a few frame buffers and then tries to set it back to the default framebuffer.
glBindFramebuffer(GL_FRAMEBUFFER, 0);
This generates an GL_INVALID_ENUM. Only on iOS.
I am completely stumped as to why. The code runs fine on all major platforms except iOS. I'm wanting to blame GLKit. Any examples of iOS OpenGL setup that do not use GLKit?
UPDATE
The following snippet of code lets me see the default framebuffer that GLKit is using. For some reason it comes out as "2". Sure enough if I use "2" in all my glBindFrameBuffer calls it works. This is very frustrating.
[view bindDrawable ];
GLint defaultFBO;
glGetIntegerv(GL_FRAMEBUFFER_BINDING_OES, &defaultFBO);
LOGI("DEFAULT FBO: %d", defaultFBO);
What reason on earth would cause GLKit to not generate its internal framebuffer at 0? This is the semantic all other implementations of OpenGL use, 0 is the default FBO.
On iOS there is no default framebuffer. See Framebuffer Objects are the Only Rendering Target on iOS. I don't know much about GLKit, but on iOS to render something on screen you need to create framebuffer, and attach to it renderbuffer, and inform Core Animation Layer that this renderbuffer will be the "screen" or "default framebuffer" to draw to. Meaning - everything you'll draw to this framebuffer, will appear on screen. See Rendering to a Core Animation Layer.
I feel it's necessary to point out here that the call to glBindFramebuffer(GL_FRAMEBUFFER, 0);
does not return rendering to the main framebuffer although it would appear to work for machines that run Windows, Unix(Mac) or Linux. Desktops and laptops have no concept of a main default system buffer. This idea started with handheld devices. When you make an openGL bind call with zero as the parameter then what you are doing is setting this function to NULL. It's how you disable this function. It's the same with glBindTexture(GL_TEXTURE_2D, 0);
It is possible that on some handheld devices that the driver automatically activates the main system framebuffer when you set the framebuffer to NULL without activating another. This would be a choice made by the manufacturer and is not something that you should count on, this is not part of the openGL ES spec. For desktops and laptops, this is absolutely necessary since disabling the framebuffer is required to return to normal openGL rendering.
On an iOS device, you should make the following call,
glBindFramebuffer(GL_FRAMEBUFFER, viewFramebuffer);,
providing that you named your system framebuffer 'viewFramebuffer'. Look for through your initialization code for the following call,
glGenFramebuffers(1, &viewFramebuffer);
Whatever you have written at the end there is what you bind to when returning to your main system buffer.
If you are using GLKit then you can use the following call,
[((GLKView *) self.view) bindDrawable]; The 'self.view' may be slightly different depending on your particular startup code.
Also, for iOS, you could use, glBindFramebuffer(GL_FRAMEBUFFER, 2); but this is likely not going to be consistent across future devices released by Apple. They may change the default value of '2' to be '3' or something else in the future so you'd want to use the actual name instead of an integer value.
(void)glkView:(GLKView *)view drawInRect:(CGRect)rect {
//your use secondary/offscreen frame buffer to store render result
[self.shader drawOffscreenOnFBO];
//back to default frame buffer of GLKView
[((GLKView *) self.view) bindDrawable];
//draw on main screen
[self.shader drawinmainscreen];
}
refernece .....http://districtf13.blogspot.com/
Alright, so I wrote a custom VMR9 Allocator/Presenter which seems to work fine. However, when I attempt to copy video frames from the Allocator/Presenter surfaces into my application surfaces, the video appears to flicker. Audio playback is fine so I'm fairly certain it's not an issue of the machine being bogged down or anything. This is the code I have in my render loop.
g_pd3dDevice->Clear(0, NULL, D3DCLEAR_TARGET, D3DCOLOR_ARGB(255, 0, 0, 0), 1.0f, 0);
// render the scene
if (SUCCEEDED(g_pd3dDevice->BeginScene()))
{
//g_pd3dDevice->SetRenderTarget(0, g_pd3dSurface);
g_pd3dDevice->StretchRect(vmr9_ap->renderSurface, src, g_pd3dSurface, dest, D3DTEXF_NONE);
// end the scene
g_pd3dDevice->EndScene();
}
However, if I change it to this (commenting out clearing the buffer)
// g_pd3dDevice->Clear(0, NULL, D3DCLEAR_TARGET, D3DCOLOR_ARGB(255, 0, 0, 0), 1.0f, 0);
// render the scene
if (SUCCEEDED(g_pd3dDevice->BeginScene()))
{
//g_pd3dDevice->SetRenderTarget(0, g_pd3dSurface);
g_pd3dDevice->StretchRect(vmr9_ap->renderSurface, src, g_pd3dSurface, dest, D3DTEXF_NONE);
// end the scene
g_pd3dDevice->EndScene();
}
this flickering goes away. I'm worried that this is somehow bad form/hackish and might cause more problems than it solves. Does anyone have any experience in this area? Is there a better solution?
Thanks!
If you intend to repaint the entire viewport every frame there is no reason to do a clear, and it can actually yield a lot of performance gains, so go for it! As for your flickering, that might be something different. Are you doing you're drawing in a WM_PAINT message? If so, you may want to also intercept the WM_ERASEBKGND message and simply return 1 when you get it. This prevents windows from trying to erase the background and has helped me get rid of some flickering in the past.
FYI: Ever do the noclip cheat in Doom or Quake, and when you walk outside a wall everything starts leaving "trails" all over? That's because they're not clearing the back buffer, since under normal circumstances the entire scene would be redrawn every time anyway. I say if it's good enough for id it's good enough for me! :)
Edit: Oh, and on more thing! I'm not sure if it's required or not, but I always do my clear AFTER calling BeginScene(). Could also be contributing to your flicker.
TBH i reckon you are best off writing your own directshow render filter that copies the data directly to a texture and then drawing a quad over the screen with the texture. You'd get much better performance. Writing a render filter is actually pretty easy. Especially when you appreciate you don't have to expose it to the operating system so most of the difficult DirectShow hurdles don't need t be jumped.
Edit: Look up the "Dump Filter" it comes as part of Microsoft's DirectShow helper code ...
I was faced with the same problem. In my case, the flickering reason was in StretchRect call inside of a BeginScene/EndScene pair.