I am wondering if anyone has any good patterns for handling lighting in XNA for the Xbox 360. I am currently trying to implement a spotlight component. I have read about deferred shading as a means of reducing complexity. I have also heard that this is not a good idea for the Xbox 360 and that it is more difficult in XNA 4.0 than it was in XNA 3.0. My goal is to make a reusable component that encapsulates a spotlight effect that can be used through a game.
Thanks,
-John
Deferred Rendering allows you to apply lighting after the world itself has been rendered.
It does reduce complexity in the sense that you are not forced to choose which lights will affect a model before drawing it (usually the closest/most significant ones).
It can make translucency handling and anti-aliasing more complex.
It allows for many lights to be applied, and move around the scene, something that is impractical with forward lighting.
It requires several buffers to be present - a geometry buffer as a minimum, but normal and specular buffers are helpful also. Techniques exist to pack such buffers, but they do require more memory, and use bandwidth to write to. This means that deferred rendering has a greater up front cost then forward rendering does, but provides more flexibility in relation to the amount of lights, and moving them.
Starcraft 2, Gears of War and other games use a deferred renderer, so its certainly a proven solution.
It is not very different in terms of complexity under XNA 4 here is a link showing an implementation under XNA 4. The main difference if having to explicitely store a copy of the z buffer rather than accessing it directly. http://roy-t.nl/index.php/tag/deferred-rendering/ and you could consider light pre-pass also http://jcoluna.wordpress.com/
Since you want a spotlight effect I can see why you'd consider deferred rendering, as I'm guessing that the spotlight is dynamic. However if you haven't used deferred rendering before, I'd suggest exploring Roy's link above to decide if your happy with the cost/complexity.
Related
Recently I am studying the research NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis(https://www.matthewtancik.com/nerf), and I am wondering: What is it used for? Will there be any application of NeRF?
The result of this technique is very impressive, but what is it used for? I keep thinking of this question over and over again. It is very realistic, the quality is perfect, but we don't want to see the camera swinging around all the time, right?
Personally, this technique has some limitations:
Cannot generate views that never seen in input images. This technique interpolates between two views.
Long training and rendering time: According to the authors, it takes 12 hours to train a scene, and 30s to render one frame.
The view is static and not interactable.
I don't know if it is appropriate to compare NeRF with Panorama and 360° image/video, essentially they are different, only NeRF uses deep learning to generate new views, the others basically are just using smart phone/camera to capture scenes plus some computer vision techniques. Still, the long training time makes NeRF less competitive in this application area. Am I correct?
Another utility I can think of is product rendering, however, NeRF doesn't show advantages compare to using 3D software to render. Like commercial advertisement, usually it requires animation and special effects, then definitely 3D software can do better.
The potential use of NeRF might be 3D reconstruction, but that would be out of the scope, although it is able to do that. Why do we need to use NeRF for 3D reconstruction? Why not use other reconstruction techniques? The unique feature of NeRF is the ability of creating photo-realistic views, if we use NeRF for 3D reconstruction, then this feature becomes pointless.
Does anyone have new ideas? I would like to know.
Why do we need to use NeRF for 3D reconstruction?
The alternative would be multi-view stereo, which produces point clouds of finite resolution and is susceptible to illumination changes. If you then render such point cloud without non-trivial post-processing, it will not look photorealistic.
I don't know if it is appropriate to compare NeRF with Panorama and 360° image/video,
Well, if you deal with exactly flat scene with simple lighting (i.e. ambient light and Lambertian objects), then you can use panorama techniques for new view synthesis. In general though, it won’t produce the result you expect. You have to know the depth to interpolate correctly.
When it comes to practical limitations (slow; does not model deformations), NeRF should be considered a milestone that provided a proof of concept that representing surface as a level set of MLP-modelled function can result in sharp rendering. There is already good progress in addressing those limitations, and multiple works apply this idea for practical tasks.
I have been maintaining my own custom 2D library -written in Objective-C / OpenGL ES 2.0- for a while now, to use in my personal projects (not work). I have also tried cocos2d and SpriteKit now and then, but eventually settled for "reinventing the wheel" because
It's fun,
Knowledge-wise, I'd rather be the guy who can code a graphics library than just a guy who can use one,
Unlimited possibilities for customization.
Now, I am transitioning my code base to Swift and (besides all the design differences that arise when moving to a language where class inheritance takes a back seat to protocols, etc) I was thinking that while I'm at it, I should consider transitioning to Metal as well. If anything, for the sake of future-proofness (also, I'm all for learning new technologies, and to be sincere OpenGL/OpenGL ES are a terribly cluttered bag of "legacy" and backwards compatibility).
My library is designed around all sorts of OpenGL (ES)-specific performance bottlenecks: Use of texture atlases and mesh consolidation to reduce draw calls, rendering opaque sprites first, and semitransparent ones last (ordered back to front), etc.
My question is: Which of these considerations still apply to Metal, and which ones should I not even bother implementing (because they're not a performance issue anymore)?
Metal is only available on the subset of IOS devices which support OpenGLES3, so te be fair you need to compare Metal to GLES3.
Texture atlases & mesh consolidation:
With Metal, CPU cost of draw calls is lower than with GLES3, and you can parallelize draw call setup on multiple threads.
So this could allow you to skip atlasing & consolidation ... but those are good practices so it would be even better if you kept those with Metal and use the extra CPU time to do more things !
Note that with GLES3 by using instancing & texture arrays you should also be able to get rid of atlasing and keep a low draw call count.
Rendering opaque sprites first, and semitransparent ones last
Metal will change absolutely nothing to this, this is a constraint from the PowerVR GPUs tile based defered renderer, whatever driver you use this will not change the GPU hardware. And anyway rendering opaques before semi transparent is the recommended way to proceed when you do 3D, wheter you use DirectX, OpenGL or Metal ...
Metal will not help if you are fillrate bound !
In general, Metal will only give you improvements on the CPU side.
If your performance is limited by fillrate (fragment shaders too complex, too much transparent overdraw, resolution too high etc.) then you will get the exact same result in Metal and GLES3 (assuming that you have carefully optimized shaders for each platform).
I have just delved into the world of Metal, and I thought that I'd got the hang of it! But then it occurred to me that if I wanted to make a game, then static objects moving around a screen wouldn't suffice. So my question is, 'Is it possible to create animations for models with Metal?'
I have looked at using other APIs, such as SpriteKit, and SceneKit, but I found that they do not support shaders, and are not as powerful as Metal.
The only way that I can think about how I would go about this, is by creating 60 different models, and then loading each one one after the other, to give a 'stop-motion' kind of effect, but I think that this would probably be incredibly inefficient, and was hoping that there was an easier answer?
Thanks a lot!
Yes, there are other, more efficient ways to do animation. But before getting into that, a warning: it really looks like you're barking up the wrong tree here.
Metal is a (conceptually) very low-level interface. You use Metal to talk (almost) directly to the GPU, so to work with it you need to think (sort of) like a GPU: in terms of data buffers, vertex transformations, etc. You seem to be working at a much higher conceptual level, so you're probably better served by one of the high-level game engines: SpriteKit for 2D or SceneKit for 3D. (Or a third party engine like Cocos or Unity.) Metal, on the other hand, is better suited for building those game engines.
SpriteKit and SceneKit do support shaders. Look at SKShader and SCNShadable in the docs (and be sure to click the "More" links to read the full overviews). SceneKit also supports character animations (aka skeletal animation aka skinning): typically one designs and rigs a model for animation in an external authoring tool (Maya, Blender, etc), then uses SceneKit to work with the animations at run time.
It is possible to do things like GPU-based skeletal animation in Metal. But I haven't seen any tutorials or similar written about it yet, probably because Metal is such a new technology. Fundamentally, though, it'd be based on the same sorts of techniques you'd use for skeletal animation in OpenGL or Direct3D — and much has been written about animation for those technologies. If you're willing to invest the time and energy to work at a low level, adapting the subject matter from GL/D3D tutorials is relatively easy.
You can do skeletal animation in Metal, SCNKit would be using the GPU to deform the mesh as well. But to do it in Metal you would need to pass skin weights, along with bone matrices for the bind pose and the transformations of the bones as they animate then calculate the new vertex positions based on these. In fact I think you need the inverse of the bind pose matrices. Each mesh vertex is then transformed by a weighted sum of transformations dictated by the skin weights.
I tried it but screwed it up somehow it didn’t deform properly, I don’t know if I’d obtained the wrong matrices from my custom script to grab animation data from blender or a bug in my shader maths or from the weights.
It was probably close, but with all the possible things that I may have got wrong in the process it was difficult to fix so I abandoned it in the end.
Probably easier to stick with SceneKit and let apple take care of the rest or use an existing game engine such as Unity.
Then again if you want a challenge, I’m sure it’s possible, just a little tricky. You could try CPU first to make sure the maths is ok then port it to the GPU to make it faster?
SceneKit do support shaders. And an object that manages the relationship between skeletal animations and the nodes and geometries they animate is SCNSkinner from SceneKit.
Typically, you need to create a skinned model using, for example, Autodesk Maya, save it along with animations that use the skeleton, in a scene file. You load the model from the scene file and pose or animate it in your app, either by using animation objects also loaded from the scene file or by directly manipulating the nodes in the skeleton. That's it.
Watch this 7-parts video about Blender's skeletal system and how to use it in SceneKit.
convenience init(baseGeometry: SCNGeometry?, //character
bones: [SCNNode], //array of bones
boneInverseBindTransforms: [NSValue]?, //ibt of matrix4
boneWeights: SCNGeometrySource, //influence on geometry
boneIndices: SCNGeometrySource //index mapping
)
Are these classes supported in Stage3D? Or are there equivalents or similar classes that exist?
flash.display.BitmapData;
flash.display.GraphicsSolidFill;
flash.display.GraphicsStroke;
flash.display.GraphicsPath;
flash.display.IGraphicsData;
flash.display.Shape;
flash.filters.BlurFilter;
flash.geom.ColorTransform;
Stage3D is an entirely different, fairly low-level beast. Those classes you list there are all related to the traditional Flash DisplayList, which is a CPU-driven rendering engine, so no, they don't exist, per se. But there's much more to it than that:
If you're using the raw Stage3D APIs (example tutorial here), then it feels very much like OpenGL programming. You're loading Vertex buffers, Index buffers, and textures into the GPU, and defining Vertex and fragment shader programs in an assembly language called AGAL. All this gets you a cross-platform, hardware accelerated application that's probably very fast, but it's very different than the traditional Flash DisplayList. Can you get gradients, filters and vector shapes - sure, but probably with custom shaders and such, not using those classes.
In some applications, it makes sense to use the traditional DisplayList for interactive UI controls on top of the Stage3D hardware accelerated backdrop. The DisplayList sits on top of the Stage3D plane, so this is entirely possible.
However, if such low-level 3D programming is not what you're interested in, you can choose to build on top of a framework. There are many Stage3D frameworks - some are intended for creating 3D applications, others are intended for 2D (but using the underlying 3D acceleration for speed). Adobe has a list of these frameworks here.
For example, Starling is a Stage3D framework that's intended to mimic the traditional Flash DisplayList, so it'll get you close to some of the classes you've mentioned above - check out their demo and API docs for specifics.
Another technique that Flash enables is blitting, which generates Bitmaps for 3D acceleration on the fly. You can draw into Bitmaps (aka blit) any Flash DisplayObjects you like (Shapes, drawn gradients, with filters, whatever), then push those Bitmaps into the 3D acceleration framework. You can blit individual objects separately, or blit the entire stage into one full-screen texture using this technique. But you have to be careful how often and how much you upload new textures into the GPU, because this can affect performance significantly. In fact, a significant performance consideration in GPU programming is the ability to batch several bitmaps into a single texture.
So there are many facets to consider when thinking about transitioning from the traditional DisplayList to Stage3D. Hope this helps. :)
I understand that cocos2d it's really simple API, and that I can use it to do simple and huge 2D or even sometimes 3D games/applications. As well I understand that OpenGL it's more complicated, it's lower level API etc.
Question: What is better for implementing 2D/3D games? Why do we need to learn OpenGL if we have simple frameworks like cocos2d? What you can do with OpenGL that you can't do with cocos2d?
Thanks in advance!
What is better for implementing 2D/3D games?
Hard to tell, but a higher level API is always there to make things easier for you. For example you are writing a 2D shootem up. You will likely use a game loop, you will want to use sprites and make those move on the screen. You may want animations like explosions taking place. You'll end up writing your own higher level API to do those things. Cocos2D has solved those problems for you already. Any other frameworld should have solved it.
Why do we need to learn OpenGL if we have simple frameworks like cocos2d?
In case you like to cusomize the standard behaviour of a framework, especially the drawing part, you should get into openGL. If there is something you like to have which doesn't come out of the box you may find yourself reimplementing a base framework class. For example, look at the shaders used in Cocos2D 2.0. If you like some special blending mode, like a tinting effect, you won't get it for free. There is a colour attribute for a CCSprite but this may not be the result you're expecting. So you'll have to write your own shader and plug it into the sprite you like to be displayed in a different way.
What you can do with OpenGL that you can't do with cocos2d?
This comparison doesn't really work out, since cocos2d facilitates opengGL for the drawing part to build up that higher level api and make your life easier as a game developer.
Cocos2d is a wrapper around the 2D features of OpenGL (as of this: http://www.cocos2d-iphone.org/about) . Under the hood it itself uses OpenGL ES to implement its features. This is good because it means that there will be minimal performance overhead so you can start using its simpler API without having to get immersed to the definitely bigger learning path of OpenGL.
It has however only strong 2D support and if you plan to write later 3d games you loose all benefits of Cocos2d: why would you rewrite a 3d rendering engine with a 2d framework that under the hood uses a very strong 3d engine? You loose performance for a lot of unnecessary work.
So the simpler answer is: for 2d Cocos2d, for 3d OpenGL.
If you want to start OpenGL ES, this is a very good tutorial for beginners: http://iphonedevelopment.blogspot.it/2009/05/opengl-es-from-ground-up-table-of.html