I am able to render the DirectX and SDL content on the SDL Window but when the content of DirectX is rendered it hides the content of SDL and when the content of SDL is rendered it hides the content of DirectX. This is because the DirectX and SDL buffers are different.
is there any way that i can store the two contents in a single buffer(either of DirectX or SDL)?or is there any way to cope up with?
You'll probably need to render DirectX content to a texture, copy the pixels over to an SDL texture, and do your SDL rendering with that.
http://www.rastertek.com/dx10tut22.html can help with rendering DirectX to a texture, and you can use SDL_Surface objects (see SDL_CreateRGBSurface and friends) for the SDL side. Be careful with pixel formats as well.
You can also go the reverse direction (which may be easier). Since you're already given an SDL_Surface object for the main rendering buffer, you can copy pixel data from that into a new DirectX texture.
Hope this helps - I can post any concrete code later if need be.
Related
I'm using webgl to do YUV to RGB conversions on a custom video codec.
The video has to play at 30 fps. In order to make this happen I'm doing all my math every other requestAnimationFrame.
This works great, but I noticed when profiling that uploading the textures to the gpu takes the longest amount of time.
So I uploaded the "Y" texture and the "UV" texture separately.
Now the first "requestAnimationFrame" will upload the "Y" texture like this:
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, yTextureRef);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, textureWidth, textureHeight, 0, gl.LUMINANCE, gl.UNSIGNED_BYTE, yData);
The second "requestAnimationFrame" will upload the "UV" texture in the same way, and make a draw call to the fragment shader doing the math between them.
But this doesn't change anything in the profiler. I still show nearly 0 gpu time on the frame that uploads the "Y" texture, and the same amount of time as before on the frame that uploads the "UV" texture.
However if I add a draw call to my "Y" texture upload function, then the profiler shows the expected results. Every frame has nearly half the gpu time.
From this I'm guessing the Y texture isn't really uploaded to the gpu using the texImage2d function.
However I don't really want to draw the Y texture on the screen as it doesn't have the correct UV texture to do anything with until a frame later. So is there any way to force the gpu to upload this texture without performing a draw call?
Update
I mis-understood the question
It really depends on the driver. The problem is OpenGL/OpenGL ES/WebGL's texture API really sucks. Sucks is a technical term for 'has unintended consequences'.
The issue is the driver can't really fully upload the data until you draw because it doesn't know what things you're going to change. You could change all the mip levels in any order and any size and then fix them all in between and so until you draw it has no idea which other functions you're going to call to manipulate the texture.
Consider you create a 4x4 level 0 mip
gl.texImage2D(
gl.TEXTURE_2D,
0, // mip level
gl.RGBA,
4, // width
4, // height
...);
What memory should it allocate? 4(width) * 4(height) * 4(rgba)? But what if you call gl.generateMipmap? Now it needs 4*4*4+2*2*4+1*1*4. Ok but now you allocate an 8x8 mip on level 3. You intend to then replace levels 0 to 2 with 64x64, 32x32, 16x16 respectively but you did level 3 first. What should it do when you replace level 3 before replacing the levels above those? You then add in levels 4 8x8, 5 as 4x4, 6 as 2x2, and 7 as 1x1.
As you can see the API lets you change mips in any order. In fact I could allocate level 7 as 723x234 and then fix it later. The API is designed to not care until draw time when all the mips must be the correct size at which point they can finally allocate memory on the GPU and copy the mips in.
You can see a demonstration and test of this issue here. The test uploads mips out of order to verify that WebGL implementations correctly fail with they are not all the correct size and correctly start working once they are the correct sizes.
You can see this was arguably a bad API design.
They added gl.texStorage2D to fix it but gl.texStorage2D is not available in WebGL1 only WebGL2. gl.texStorage2D has new issues though :(
TLDR; textures get uploaded to the driver when you call gl.texImage2D but the driver can't upload to the GPU until draw time.
Possible solution: use gl.texSubImage2D since it does not allocate memory it's possible the driver could upload sooner. I suspect most drivers don't because you can use gl.texSubImage2D before drawing. Still it's worth a try
Let me also add that gl.LUMIANCE might be a bottleneck as well. IIRC DirectX doesn't have a corresponding format and neither does OpenGL Core Profile. Both support a RED only format but WebGL1 does not. So LUMIANCE has to be emulated by expanding the data on upload.
Old Answer
Unfortunately there is no way to upload video to WebGL except via texImage2D and texSubImage2D
Some browsers try to make that happen faster. I notice you're using gl.LUMINANCE. You might try using gl.RGB or gl.RGBA and see if things speed up. It's possible browsers only optimize for the more common case. On the other hand it's possible they don't optimize at all.
Two extensions what would allow using video without a copy have been proposed but AFAIK no browser as ever implemented them.
WEBGL_video_texture
WEBGL_texture_source_iframe
It's actually a much harder problem than it sounds like.
Video data can be in various formats. You mentioned YUV but there are others. Should the browser tell the app the format or should the browser convert to a standard format?
The problem with telling is lots of devs will get it wrong then a user will provide a video that is in a format they don't support
The WEBGL_video_texture extensions converts to a standard format by re-writing your shaders. You tell it uniform samplerVideoWEBGL video and then it knows it can re-write your color = texture2D(video, uv) to color = convertFromVideoFormatToRGB(texture(video, uv)). It also means they'd have to re-write shaders on the fly if you play different format videos.
Synchronization
It sounds great to get the video data to WebGL but now you have the issue that by the time you get the data and render it to the screen you've added a few frames of latency so the audio is no longer in sync.
How to deal with that is out of the scope of WebGL as WebGL doesn't have anything to do with audio but it does point out that it's not as simple as just giving WebGL the data. Once you make the data available then people will ask for more APIs to get the audio and more info so they can delay one or both and keep them in sync.
TLDR; there is no way to upload video to WebGL except via texImage2D and texSubImage2D
How can I resize texture2d of SharpDX? I'm using SharpDX to Duplicate the screen and I use MediaFoundation to encode the texture into a video file. My problem is when I open an application into fullscreen and has a different resolution from system resolution I got a blank screen on my recording. Is there a better way I can resize the texture before encoding to mediafoundation without suffering performance? I'm using hardware accelerated. Thanks.
It depends on how exactly you use Media Foundation, but you can use the Video Processor MFT explicitly. You need to add this MFT to the topology if you use IMFMediaSession/IMFTopology. Or initialize and process the samples with this MFT if you use Sink Writer. In this case you need to supply the DX manager to the MFT, using MFT_MESSAGE_SET_D3D_MANAGER.
This MFT is only available on Windows 8 and higher. So if you need to support an older Windows version you can use Video Resizer, but it is not hardware accelerated.
Another option to resize the texture would be to create a render target of the desired size and draw the texture to it. After that you need to feed that render target to the encoder.
This question's kind of a 4-parter:
Is it true that all BitmapData is immediately cached to the GPU as soon as it's created (even if it's never applied to a Bitmap or added to stage?)
Does this still happen if the GPU texture buffer is already full? Bonus points: if so, what's the preferential swap method the GPU chooses to select which textures to remove from memory?
If (1), then does setting the width/height of any BitmapData uncache it and/or does replacing its pixels therefore upload the new pixels to the same memory address on the GPU? Bonus: What if the size changes?
To bring this all together, would a hybrid class that extends BitmapData but stores its actual data in a ByteArray be able to use setPixels/getPixels on itself to control upload/download from the GPU as necessary, to buffer a large number of bitmaps? Bonus: Would speed improve for actually placing them in Bitmaps if the instances of this class were static?
Here are some answers
No. In AIR, you manually upload bitmaps to GPU and have control WHEN to do it
As far as I've reached, if the buffer is full, you simply get an error for it - the GPU cannot make a choice what do to. Removing a random texture won't be nice if it's important to you, right? :)
You can check for example Starling and how it uploads textures to GPU. Once you force it to do so, it doesn't care what you do with the bitmap. It's like making a photo image of an object so that you can just show it instead of explaining it with words. It won't matter if you change the object, the photo will be still the same.
Simplified answer: no. Again - it's best to check out how textures are created and how you upload stuff to GPU.
Does anyone know if it is possible in XNA to dynamically load bitmap from jpg files and draw texture2d using other surface formats then SurfaceFormat.Color?
Because of memory/speed limits I need to use Bgr565 or Dxt surface formats.
I could use content pipeline and set Content Processor, Texture Format to DxtCompressed but Content.Load doesn't allow dynamic texture load.
I could use Texture2D.FromStream but it creates SurfaceFormat.Color texture only.
I experiment with creating empty Texture2D in desired surface format and try to manualy set bitmap data using SetData but it seems like a too complicated way.
Is there Texture2D.FromStream and Texture2D.SetData the only way to dynamically load jpg files and use them as Bgr565 or Dxt surface format textures?
Haven't got a solution for loading with SurfaceFormat. But you can load them and draw them to a rendertarget with the desired format, then copy the textureData to a new texture you create with the required format?
Will increase loading-times, but should allow the sped-up execution you are looking for?
I have a directx9 game engine that creates its normal adaptor with this format:
D3DFMT_X8R8G8B8
I have a system where I render some objects to an offscreen render target, as lightmaps. I then use that lightmap data to composite back to the back buffer where they act as a full screen 'mask' and let me get the effect of torches or other light sources on a dark scene.
Everything works just great.
The problem is, I'm aware that my big offscreen lightmap render targets are 16MB each, at a large res, and I only really need 8 bits of data (greyscale) from them, so 75% of the 32 bit render target memory is a waste. (I'm targeting low spec cards).
I tried creating the render targets as
D3DFMT_A8
But directx silently fails on that (if I add CheckDeviceFormat() I see it happen) and creates 32 bit anyway. I use the D3DXCreateTexture function
My question is, what format is best for creating these offscreen buffers?
Thankyou for your help, I'm not good at render target related stuff :)
D3DFMT_L8 is 8 bit luminance. I believe it's supported on GeForce 3 (i.e. the first consumer card with shader 1.1!), so must be available everywhere. I think the colour is read as L, L, L, 1, i.e. rgb = luminance value, alpha = 1.
Edit: this tool is useful for finding caps:
http://zp.lo3.wroc.pl/cdragan/wizard.php
Ontopic: If you are targeting lower spec cards, you are very likely to be running on systems where 8-bit single channel render targets are not supported at all.
If you are using shaders to do the rendering and compositing, it should be possible to use the rgba channels for 4 alternating pixels of your lightmap, packing your information. Perhaps you can tell us a little bit more about your current rendering setup?
Offtopic: AWESOME to have you here on StackOverflow, big fan of your work!