I want to use Electron as a debug overlay for a Vulkan Render Engine im building. Since i have a lot of requirements on this debug tool writing one in engine myself would take way too long. I would like to use electron instead of Qt or similar since i feel its a lot more powerful and flexible with less effort (once its working).
The problem is now that i somehow either have to get my render output to electron or electrons output to my engine. As far as i can tell the easiest solution would be to copy the data back to cpu then transfer it. But that would be extremely slow and cost a lot of bandwidth. So i was wondering if there is a better solution.
I have two ideas to make it work but i didnt find any ways to implement them or even anyone talking about it.
The first would be to have electron configured to run on the gpu somehow get the handle for the output texture and importing it into my render engine using vulkan external memory. However as i have no experience with chromium and there doesnt seem to be anyone else that did it this i dont think it would work out to well.
The second idea was to do the opposite. Using a canvas element with webgl and again using vulkan external memory to copy the output of my engine to a texture and displaying it. I have full control over the draw process here so i think it would be a lot simpler and more stable. However again i found no way of setting up a webGL texture handle as an external memory object.
Is there any better way of doing this or some help on how to implement it?
Related
I am a mathematician and not a programmer, I have a notion on the basics of programming and am a quite advanced power-user both in linux and windows.
I know some C and some python but nothing much.
I would like to make an overlay so that when I start a game it can get info about amd and nvidia GPUs like frame time and FPS because I am quite certain the current system benchmarks use to compare two GPUs is flawed because small instances and scenes that bump up the FPS momentarily (but are totally irrelevant in terms of user experience) result in a higher average FPS number and mislead the market either unintentionally or intentionally (for example, I cant remember the name of the game probably COD there was a highly tessellated entity on the map that wasnt even visible to the player which lead AMD GPUs to seemingly under perform when roaming though that area leading to lower average FPS count)
I have an idea on how to calculate GPU performance in theory but I dont know how to harvest the data from the GPU, Could you refer me to api manuals or references to help me making such an overlay possible?
I would like to study as little as possible (by that I mean I would like to learn what I absolutely have to learn in order to get the job done I dont intent to become a coder).
I thank you in advance.
It is generally what the Vulkan Layer system is for, which allows to intercept API commands and inject your own. But it is nontrivial to code it yourself. Here are some pre-existing open-source options for you:
To get to timing info and draw your custom overlay you can use (and modify) a tool like OCAT. It supports Direct3D 11, Direct3D 12, and Vulkan apps.
To just get the timing (and other interesting info) as CSV you can use a command-line tool like PresentMon. Should work in D3D, and I have been using it with Vulkan apps too and it seems to accept them.
My goal is as follows: I have to read in a video that is stored on the sd card, process it frame for frame and then store it in a new file on the SD card again，In each image to do image processing.
At first I wanted to use opencv for android but I did not seem to be able to read the video
here.
I am guessing you already know that doing this on a mobile device or any compute limited devices is not ideal, simply because video manipulation is very computer intensive which translates to slow execution and heavy battery usage on many devices. If you do have the option to do the processing on the server side it is definitely worth considering.
Assuming that for your use case you need to do it on the mobile device, then OpenCV on Android will now allow you to read in a video and access each frame - #StephenG mentions this in his answer to the question you refer to above.
In the past, functionality like this did not get ported to the Android OpenCv as the guidance was to use ffmpeg for frame grabbing on Android devices.
According to more recent documentation, however, this should be available for Android now using the VideoCapture class (note I have not used this myself...):
http://docs.opencv.org/java/2.4.11/org/opencv/highgui/VideoCapture.html
It is worth noting that OpenCV Android examples are all currently based around Eclipse and if you want to use Studio, getting things up an running initially can be quite tricky. The following worked for me recently, but as both studio and OpenCV can change over time you may find you have to do some forum hunting if it does not work for you:
https://stackoverflow.com/a/35135495/334402
Taking a different approach, you can use ffmpeg itself, in a wrapper in Android, for tasks like this.
The advantage of the wrapper approach is that you can use all the usual command line syntax and there is a lot of info on the web to help you get the right parameters.
The disadvantage is that ffmpeg was not really designed to be wrapped in this way so you do sometimes see issues. Having said that it is a common approach now and so long as you choose a well used wrapper library you should at least have a good community to discuss any issues you come across with. I have used this approach in a hand crafted way in the past but if I was doing it again I would use one of the popular examples such as:
https://github.com/WritingMinds/ffmpeg-android-java
I have a chart in my current web app that I’ve implemented in canvas. I’d like this to work in IE8, but excanvas doesn’t seem to support translucency or composite operations. My fallback solution is to render a chart on the server as an image and send that out to IE8 instead of rendering it client-side.
I’d assumed there’d be a canvas gem that I could use with a direct port of my JS code to Ruby, but I can’t find anything. Has no-one done this? If not, what would people recommend? It’s not a particularly complex drawing, but I’d like to keep the amount of duplication to a minimum.
(It’s worth pointing out that I’ve considered using a headless Webkit to render and return a data URI, but I expect this would be fairly slow to spin up. Another possibility is to pre-render all the possible charts – somewhere around 120K of them – but that feels like a last resort!).
I haven't found such an implementation.
There is at least one canvas implementation for node.js. You could use it to write a small node program to generate the images using the exact same code you're using on the client. It wouldn't be the most efficient solution but I'd guess it'd be better than using PhantomJS or the like.
Best as in reliable, maintainable and fast.
Considering Processing, VVVV or OpenFrameworks?
I know Processing doesn't handle big video frames very well.
VVVV (Nodes use OpenCV) is just for Windows.
OpenFrameworks (OpenCv) is more complicated than the
above.
You can try to implement your app in Processing and see if it fits your needs and is fast enough. It should a little more easy and faster to write Java instead of C++.
Here can you find how to setup with processing with examples: http://ubaa.net/shared/processing/opencv/
If you don't want to code anything you can try VVVV, should be little faster but only on Windows as you mentioned.
If your Processing app is running too slow, you can try openFrameworks.
download it the new OF 007 from http://www.openframeworks.cc/ and check out the setup guide.
If you have done the install you can play around with the openCV examples from
<your-OF-folder>/apps/addonsExamples/opencvExample
<your-OF-folder>/apps/addonsExamples/opencvHaarFinderExample/
Personally I prefer OF because you can do any custom thing with the most performance, but its good to make your prototype with Processing to see if it works and implement it after that again in OF.
As far as I can see from your question, VVVV and OF are the options your looking at, but you prefer VVVV's node based programming over OF, but aren't happy that VVVV is Windows only.
Have you considered other alternatives like MaxMSPJitter or PureData ?
Both are similar to VVVV or the other way around :)
MaxMSP has a package for 'optimized matrix operations'(3D/video) called Jitter.
For Jitter there is a cv.jit free collection of external objects and the samples/tutorials are great.
Similarly PureData has an add-on called Gem, which is similar to Max's Jitter package.
I haven't tried with PureData, but there are OpenCV bindings for it, through Gem.
cv.jit
pdp OpenCV PureData Bindings - via Piksel.no
MaxMSP uses quicktime on osx and can use directX on windows, but it's commercial.
PureData runs on windows/osx/linux, it's free and opensource.
HTH
I am rendering a certain scene to an off-screen frame buffer (FBO) and then I'm reading the rendered image using glReadPixels() for processing on the CPU. The processing involves some very simple scanning routines and extraction of data.
After profiling I realized that most of what my application does is spend time in glReadPixels() - more than 50% of the time. So the natural step is to move the processing to the GPU so that the data would not have to be copied.
So my question is - what would be the best way to program such a thing to the GPU?
GLSL?
CUDA?
Anything else I'm not currently aware of?
The main requirements is that it'll have access to The rendered off-screen frame bufferes (or texture data since it is possible to render to a texture) and to be able to output some information to the CPU, say in the order of 1-2Kb per frame.
You might find the answers in the "Intro to GPU programming" questions useful.
-Adam
There are a number of pointers to getting started with GPU programming in other questions, but if you have an application that is already built using OpenGL, then probably your question really is "which one will interoperate with OpenGL"?
After all, your whole point is to avoid the overhead of reading your FBO back from the GPU to the CPU with glReadPixels(). If, for example you had to read it back anyway, then copy the data into a CUDA buffer, then transfer it back to the gpu using CUDA APIs, there wouldn't be much point.
So you need a GPGPU package that will take your OpenGL FBO object as an input directly, without any extra copying.
That would probably rule out everything except GLSL.
I'm not 100% sure whether CUDA has any way of operating directly on an OpenGL buffer object, but I don't think it has that feature.
I am sure that ATI's Stream SDK doesn't do that. (Although it will interoperate with DirectX.)
I doubt that the DirectX 11 "technology preview" with compute shaders has that feature, either.
EDIT: Follow-up: it looks like CUDA, at least the most recent version, has some support for OpenGL interoperability. If so, that's probably your best bet.
I recently found this Modern GPU
You may find OpenAI Triton useful