Related
i'm trying to load image on webgl by texture, the detail steps is:
after loaded image ,create positions and bind positionBuffer:
...
const positions = [ // here width is viewport width, and height calculted by height = naturalHeight / naturalWidth * width;
-width / 2, -height / 2, 0.0,
width / 2, -height / 2, 0.0,
width / 2, height / 2, 0.0,
-width / 2, height / 2, 0.0,
]
...
create textureCoordinates and bind textureCoordBuffer
create indices( define which vertexes combines a plane ) and bind indexBuffer
create projectionMatrix ,modelViewMatrix, pull out the positions from the positionbuffer into the vertexPosition attribute, pull out the texture coordinates from the texture coordinate buffer into the textureCoord attribute
pass projectionMatrix,modelViewMatrix by gl.uniformMatrix4fv
bindTexture:
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, 1);
const level = 0;
const internalFormat = gl.RGBA;
const srcFormat = gl.RGBA;
const srcType = gl.UNSIGNED_BYTE;
const texture = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.texImage2D(gl.TEXTURE_2D, level, internalFormat, srcFormat, srcType, image);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR );
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER,gl.LINEAR );
last step is active texture0,bindTexture,bound the texture to texture unit 0, and gl.drawElements(gl.TRIANGLES, vertexCount, type, offset);
After these steps, the result on the screen just looks compressed, the image becomes fuzzy and full of sawtooth.
What could i do to solve this problem ?
There are many abstractions around WebGL for running parallel processing it seems, e.g.:
https://github.com/MaiaVictor/WebMonkeys
https://github.com/gpujs/gpu.js
https://github.com/turbo/js
But I am having a hard time understanding what a simple and complete example of parallelism would look like in plain GLSL code for WebGL. I don't have much experience with WebGL but I understand that there are fragment and vertex shaders and how to load them into a WebGL context from JavaScript. I don't know how to use the shaders or which one is supposed to do the parallel processing.
I am wondering if one could demonstrate a simple hello world example of a parallel add operation, essentially this but parallel form using GLSL / WebGL shaders / however it should be done.
var array = []
var size = 10000
while(size--) array.push(0)
for (var i = 0, n = 10000; i < n; i++) {
array[i] += 10
}
I guess I essentially don't understand:
If WebGL runs everything in parallel automatically.
Or if there is a max number of things run in parallel, so if you have 10,000 things, but only 1000 in parallel, then it would do 1,000 in parallel 10 times sequentially.
Or if you have to manually specify the amount of parallelism you want.
If the parallelism goes into the fragment shader or vertex shader, or both.
How to actually implement the parallel example.
First off, WebGL only rasterizes points, lines, and triangles. Using WebGL to do non rasterization (GPGPU) is basically a matter of realizing that the inputs to WebGL are data from arrays and the output, a 2D rectangle of pixels is also really just a 2D array so by creatively providing non graphic data and creatively rasterizing that data you can do non-graphics math.
WebGL is parallel in 2 ways.
it's running on a different processor, the GPU, while it's computing something your CPU is free to do something else.
GPUs themselves compute in parallel. A good example if you rasterize a triangle with 100 pixels the GPU can process each of those pixels in parallel up to the limit of that GPU. Without digging too deeply it looks like an NVidia 1080 GPU has 2560 cores so assuming they are not specialized and assuming the best case one of those could compute 2560 things in parallel.
As for an example all WebGL apps are using parallel processing by points (1) and (2) above without doing anything special.
Adding 10 to 10000 elements though in place is not what WebGL is good at because WebGL can't read from and write to the same data during one operation. In other words, your example would need to be
const size = 10000;
const srcArray = [];
const dstArray = [];
for (let i = 0; i < size; ++i) {
srcArray[i] = 0;
}
for (var i = 0, i < size; ++i) {
dstArray[i] = srcArray[i] + 10;
}
Just like any programming language there is more than one way to accomplish this. The fastest would probably probably be to copy all your values into a texture then rasterize into another texture, looking up from the first texture and writing +10 to the destination. But, there in is one of the issues. Transferring data to and from the GPU is slow so you need to weigh that into whether doing work on the GPU is a win.
Another is just like the limit that you can't read from and write to the same array you also can't randomly access the destination array. The GPU is rasterizing a line, point, or triangle. It's fastest at drawing triangles but that means its deciding which pixels to write to in what order so your problem also has to live with those limits. You can use points to as a way to randomly choose a destination but rendering points is much slower than rendering triangles.
Note that "Compute Shaders" (not yet part of WebGL) add the random access write ability to GPUs.
Example:
const gl = document.createElement("canvas").getContext("webgl");
const vs = `
attribute vec4 position;
attribute vec2 texcoord;
varying vec2 v_texcoord;
void main() {
gl_Position = position;
v_texcoord = texcoord;
}
`;
const fs = `
precision highp float;
uniform sampler2D u_srcData;
uniform float u_add;
varying vec2 v_texcoord;
void main() {
vec4 value = texture2D(u_srcData, v_texcoord);
// We can't choose the destination here.
// It has already been decided by however
// we asked WebGL to rasterize.
gl_FragColor = value + u_add;
}
`;
// calls gl.createShader, gl.shaderSource,
// gl.compileShader, gl.createProgram,
// gl.attachShaders, gl.linkProgram,
// gl.getAttributeLocation, gl.getUniformLocation
const programInfo = twgl.createProgramInfo(gl, [vs, fs]);
const size = 10000;
// Uint8Array values default to 0
const srcData = new Uint8Array(size);
// let's use slight more interesting numbers
for (let i = 0; i < size; ++i) {
srcData[i] = i % 200;
}
// Put that data in a texture. NOTE: Textures
// are (generally) 2 dimensional and have a limit
// on their dimensions. That means you can't make
// a 1000000 by 1 texture. Most GPUs limit from
// between 2048 to 16384.
// In our case we're doing 10000 so we could use
// a 100x100 texture. Except that WebGL can
// process 4 values at a time (red, green, blue, alpha)
// so a 50x50 will give us 10000 values
const srcTex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, srcTex);
const level = 0;
const width = Math.sqrt(size / 4);
if (width % 1 !== 0) {
// we need some other technique to fit
// our data into a texture.
alert('size does not have integer square root');
}
const height = width;
const border = 0;
const internalFormat = gl.RGBA;
const format = gl.RGBA;
const type = gl.UNSIGNED_BYTE;
gl.texImage2D(
gl.TEXTURE_2D, level, internalFormat,
width, height, border, format, type, srcData);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
// create a destination texture
const dstTex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, dstTex);
gl.texImage2D(
gl.TEXTURE_2D, level, internalFormat,
width, height, border, format, type, null);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
// make a framebuffer so we can render to the
// destination texture
const fb = gl.createFramebuffer();
gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
// and attach the destination texture
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, dstTex, level);
// calls gl.createBuffer, gl.bindBuffer, gl.bufferData
// to put a 2 unit quad (2 triangles) into
// a buffer with matching texture coords
// to process the entire quad
const bufferInfo = twgl.createBufferInfoFromArrays(gl, {
position: {
data: [
-1, -1,
1, -1,
-1, 1,
-1, 1,
1, -1,
1, 1,
],
numComponents: 2,
},
texcoord: [
0, 0,
1, 0,
0, 1,
0, 1,
1, 0,
1, 1,
],
});
gl.useProgram(programInfo.program);
// calls gl.bindBuffer, gl.enableVertexAttribArray, gl.vertexAttribPointer
twgl.setBuffersAndAttributes(gl, programInfo, bufferInfo);
// calls gl.activeTexture, gl.bindTexture, gl.uniformXXX
twgl.setUniforms(programInfo, {
u_add: 10 / 255, // because we're using Uint8
u_srcData: srcTex,
});
// set the viewport to match the destination size
gl.viewport(0, 0, width, height);
// draw the quad (2 triangles)
const offset = 0;
const numVertices = 6;
gl.drawArrays(gl.TRIANGLES, offset, numVertices);
// pull out the result
const dstData = new Uint8Array(size);
gl.readPixels(0, 0, width, height, format, type, dstData);
console.log(dstData);
<script src="https://twgljs.org/dist/4.x/twgl-full.min.js"></script>
Making a generic math processor would require a ton more work.
Issues:
Textures are 2D arrays, WebGL only rasterizes points, lines, and triangles so for example it's much easier to process data that fits into a rectangle than not. In other words if you have 10001 values there is no rectangle that fits an integer number of units. It might be best to pad your data and just ignore the part past the end. In other words a 100x101 texture would be 10100 values. So just ignore the last 99 values.
The example above using 8bit 4 channel textures. It would be easier to use 8bit 1 channel textures (less math) but also less efficient since WebGL can process 4 values per operation.
Because it uses 8bit textures it can only store integer values from 0 to 255. We could switch the texture to 32bit floating point textures. Floating point textures are an optional feature of both WebGL (you need to enable extensions and check they succeeded). Rasterizing to a floating point texture is also an optional feature. Most mobile GPUs as of 2018 do not support rendering to a floating point texture so you have to find creative ways of encoding the results into a format they do support if you want your code to work on those GPUs.
Addressing the source data requires math to convert from a 1d index to a 2d texture coordinate. In the example above since we are converting directly from srcData to dstData 1 to 1 no math is needed. If you needed to jump around srcData you'd need to provide that math
WebGL1
vec2 texcoordFromIndex(int ndx) {
int column = int(mod(float(ndx),float(widthOfTexture)));
int row = ndx / widthOfTexture;
return (vec2(column, row) + 0.5) / vec2(widthOfTexture, heighOfTexture);
}
vec2 texcoord = texcoordFromIndex(someIndex);
vec4 value = texture2D(someTexture, texcoord);
WebGL2
ivec2 texcoordFromIndex(someIndex) {
int column = ndx % widthOfTexture;
int row = ndx / widthOfTexture;
return ivec2(column, row);
}
int level = 0;
ivec2 texcoord = texcoordFromIndex(someIndex);
vec4 value = texelFetch(someTexture, texcoord, level);
Let's say we want to sum every 2 numbers. We might do something like this
const gl = document.createElement("canvas").getContext("webgl2");
const vs = `
#version 300 es
in vec4 position;
void main() {
gl_Position = position;
}
`;
const fs = `
#version 300 es
precision highp float;
uniform sampler2D u_srcData;
uniform ivec2 u_destSize; // x = width, y = height
out vec4 outColor;
ivec2 texcoordFromIndex(int ndx, ivec2 size) {
int column = ndx % size.x;
int row = ndx / size.x;
return ivec2(column, row);
}
void main() {
// compute index of destination
ivec2 dstPixel = ivec2(gl_FragCoord.xy);
int dstNdx = dstPixel.y * u_destSize.x + dstPixel.x;
ivec2 srcSize = textureSize(u_srcData, 0);
int srcNdx = dstNdx * 2;
ivec2 uv1 = texcoordFromIndex(srcNdx, srcSize);
ivec2 uv2 = texcoordFromIndex(srcNdx + 1, srcSize);
float value1 = texelFetch(u_srcData, uv1, 0).r;
float value2 = texelFetch(u_srcData, uv2, 0).r;
outColor = vec4(value1 + value2);
}
`;
// calls gl.createShader, gl.shaderSource,
// gl.compileShader, gl.createProgram,
// gl.attachShaders, gl.linkProgram,
// gl.getAttributeLocation, gl.getUniformLocation
const programInfo = twgl.createProgramInfo(gl, [vs, fs]);
const size = 10000;
// Uint8Array values default to 0
const srcData = new Uint8Array(size);
// let's use slight more interesting numbers
for (let i = 0; i < size; ++i) {
srcData[i] = i % 99;
}
const srcTex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, srcTex);
const level = 0;
const srcWidth = Math.sqrt(size / 4);
if (srcWidth % 1 !== 0) {
// we need some other technique to fit
// our data into a texture.
alert('size does not have integer square root');
}
const srcHeight = srcWidth;
const border = 0;
const internalFormat = gl.R8;
const format = gl.RED;
const type = gl.UNSIGNED_BYTE;
gl.texImage2D(
gl.TEXTURE_2D, level, internalFormat,
srcWidth, srcHeight, border, format, type, srcData);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
// create a destination texture
const dstTex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, dstTex);
const dstWidth = srcWidth;
const dstHeight = srcHeight / 2;
// should check srcHeight is evenly
// divisible by 2
gl.texImage2D(
gl.TEXTURE_2D, level, internalFormat,
dstWidth, dstHeight, border, format, type, null);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
// make a framebuffer so we can render to the
// destination texture
const fb = gl.createFramebuffer();
gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
// and attach the destination texture
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, dstTex, level);
// calls gl.createBuffer, gl.bindBuffer, gl.bufferData
// to put a 2 unit quad (2 triangles) into
// a buffer
const bufferInfo = twgl.createBufferInfoFromArrays(gl, {
position: {
data: [
-1, -1,
1, -1,
-1, 1,
-1, 1,
1, -1,
1, 1,
],
numComponents: 2,
},
});
gl.useProgram(programInfo.program);
// calls gl.bindBuffer, gl.enableVertexAttribArray, gl.vertexAttribPointer
twgl.setBuffersAndAttributes(gl, programInfo, bufferInfo);
// calls gl.activeTexture, gl.bindTexture, gl.uniformXXX
twgl.setUniforms(programInfo, {
u_srcData: srcTex,
u_srcSize: [srcWidth, srcHeight],
u_dstSize: [dstWidth, dstHeight],
});
// set the viewport to match the destination size
gl.viewport(0, 0, dstWidth, dstHeight);
// draw the quad (2 triangles)
const offset = 0;
const numVertices = 6;
gl.drawArrays(gl.TRIANGLES, offset, numVertices);
// pull out the result
const dstData = new Uint8Array(size / 2);
gl.readPixels(0, 0, dstWidth, dstHeight, format, type, dstData);
console.log(dstData);
<script src="https://twgljs.org/dist/4.x/twgl-full.min.js"></script>
Note the example above uses WebGL2. Why? Because WebGL2 supports rendering to R8 format textures which made the math easy. One value per pixel instead of 4 values per pixel like the previous example. Of course it also means it's slower but making it work with 4 values would have really complicated the math for computing indices or might have required re-arranging the source data to better match. For example instead of value indices going 0, 1, 2, 3, 4, 5, 6, 7, 8, ... it would be easier to sum every 2 values if they were arranged 0, 2, 4, 6, 1, 3, 5, 7, 8 .... that way pulling 4 out at a time and adding the next group of 4 the values would line up. Yet another way would be to use 2 source textures, put all the even indexed values in one texture and the odd indexed values in the other.
WebGL1 provides both LUMINANCE and ALPHA textures which are also one channel but whether or not you can render to them is an optional feature where as in WebGL2 rendering to an R8 texture is a required feature.
WebGL2 also provides something called "transform feedback". This lets you write the output of a vertex shader to buffer. It has the advantage that you just set the number of vertices you want to process (no need to have the destination data be a rectangle). That also means you can output floating point values (it's not optional like it is for rendering to textures). I believe (though I haven't tested) that it's slower than rendering to textures though.
Since you're new to WebGL might I suggest these tutorials.
To get a sort of index of the elements drawn on the screen, I've created a framebuffer that will draw objects with solid colors of type GL_R32UI.
The framebuffer I created has two renderbuffer attached. One of color and one of depth. Here is a schematic of how it was created using python:
my_fbo = glGenFramebuffers(1)
glBindFramebuffer(GL_FRAMEBUFFER, my_fbo)
rbo = glGenRenderbuffers(2) # GL_DEPTH_COMPONENT16 and GL_COLOR_ATTACHMENT0
glBindRenderbuffer(GL_RENDERBUFFER, rbo[0])
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, width, height)
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, rbo[0])
glBindRenderbuffer(GL_RENDERBUFFER, rbo[1])
glRenderbufferStorage(GL_RENDERBUFFER, GL_R32UI, width, height)
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, rbo[1])
glBindRenderbuffer(GL_RENDERBUFFER, 0)
glBindFramebuffer(GL_FRAMEBUFFER, 0)
I read the indexes with readpixel like this:
glBindFramebuffer(GL_FRAMEBUFFER, my_fbo)
glReadPixels(x, y, threshold, threshold, GL_RED_INTEGER, GL_UNSIGNED_INT, r_data)
glBindFramebuffer(GL_FRAMEBUFFER, 0)
The code works perfectly, I have no problem with that.
But for debugging, I'd like to see the indexes on the screen
With the data obtained below, how could I see the result of drawing the indices (unsigned int) on the screen?*
active_fbo = glGetIntegerv(GL_FRAMEBUFFER_BINDING)
my_indices_fbo = my_fbo
my_rbo_depth = rbo[0]
my_rbo_color = rbo[1]
## how mix my_rbo_color and cur_fbo??? ##
glBindFramebuffer(gl.GL_FRAMEBUFFER, active_fbo)
glBlitFramebuffer transfer a rectangle of pixel values from one region of a read framebuffer to another region of a draw framebuffer.
glBindFramebuffer( GL_READ_FRAMEBUFFER, my_fbo );
glBindFramebuffer( GL_DRAW_FRAMEBUFFER, active_fbo );
glBlitFramebuffer( 0, 0, width, height, 0, 0, width, height, GL_COLOR_BUFFER_BIT, GL_NEAREST );
Note, you have to be careful, because an GL_INVALID_OPERATION error will occur, if the read buffer contains unsigned integer values and any draw buffer does not contain unsigned integer values. Since the internal format of the frame buffers color attachment is GL_R32UI, and the internal format of the drawing buffer is usually something like GL_RGBA8, this maybe not works, or it even will not do what you have expected.
But you can create a frame buffer with a texture attached to its color plane an use the texture as an input to a post pass, where you draw a quad over the whole canvas.
First you have to create the texture with the size as the frame buffer:
ColorMap0 = glGenTextures(1);
glBindTexture(GL_TEXTURE_2D, ColorMap0);
glTexImage2D(GL_TEXTURE_2D, 0, GL_R32UI, width, height, 0, GL_R, GL_UNSIGNED_INT, 0);
You have to attach the texture to the frame buffer:
my_fbo = glGenFramebuffers(1)
glBindFramebuffer(GL_FRAMEBUFFER, my_fbo)
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, ColorMap0, 0);
When you have drawn the scene then you have to release the framebuffer.
glBindFramebuffer(GL_FRAMEBUFFER, 0)
Now you can use the texture as an input for a final pass. Simply bind the texture, enable 2D textures and draw a quad over the whole canvas. The quad should range from from (-1,-1) to (1,1), with texture coordinates in range from (0, 0) to (1, 1). Of course you can use a shader, with a texture sampler uniform in the fragment shader, for that. You can read the texel from the texture a write to the fragment in an way you want.
Extension to the answer
If performance is not important, then you can convert the buffer on the CPU and draw it on the canvas, after reading the frame buffer with glReadPixels. For that you can leave your code as it is and read the frame buffer with glReadPixels, but you have to convert the buffer to a format appropriate to the drawing buffer. I suggest to use the
internal format GL_RGBA8 or GL_RGB8. You have to create a new texture with the convert buffer data.
debugTexturePlane = ...;
debugTexture = glGenTextures(1);
glBindTexture(GL_TEXTURE_2D, debugTexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, width, height, 0, GL_RGB, GL_UNSIGNED_BYTE, debugTexturePlane);
From now on you have 2 possibilities.
Either you create a new frame buffer and attach the texture to its color plane
debugFbo = glGenFramebuffers(1)
glBindFramebuffer(GL_FRAMEBUFFER, debugFbo)
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, debugTexture, 0);
and you use glBlitFramebuffer as described above to copy from the debug frame buffer to the color plane.
This should not be any problem, because the internal formats of the buffers should be equal.
Or you draw a textured quad over the whole viewport. The code may look like this (old school):
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glEnable(GL_TEXTURE_2D);
glBindTexture(GL_TEXTURE_2D, debugTexture);
glBegin(GL_QUADS);
glTexCoord2f(0.0, 0.0); glVertex2f(-1.0, -1.0);
glTexCoord2f(0.0, 1.0); glVertex2f(-1.0, 1.0);
glTexCoord2f(1.0, 1.0); glVertex2f( 1.0, 1.0);
glTexCoord2f(1.0, 0.0); glVertex2f( 1.0, -1.0);
glEnd();
I have a GLKit view set up to draw a solid shape, a line and an array of points which all change every frame. The basics of my drawInRect method are:
- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect
{
glClear(...);
glBufferData(...);
glEnableVertexAttribArray(...);
glVertexAttribPointer(...);
// draw solid shape
glDrawArrays(GL_TRIANGLE_STRIP, ...);
// draw line
glDrawArrays(GL_LINE_STRIP, ...);
// draw points
glDrawArrays(GL_POINTS, ...);
}
This works fine; each array contains around 2000 points, but my iPad seems to have no problem rendering it all at 60fps.
The issue now is that I would like the lines to fade away slowly over time, instead of disappearing with the next frame, making a persistence or phosphor-like effect. The solid shape and the points must not linger, only the line.
I've tried the brute-force method (as used in Apple's example project aurioTouch): storing the data from the last 100 frames and drawing all 100 lines every frame, but this is too slow. My iPad can't render more than about 10fps with this method.
So my question is: can I achieve this more efficiently using some kind of frame or render buffer which accumulates the color of previous frames? Since I'm using GLKit, I haven't had to deal directly with these things before, and so don't know much about them. I've read about accumulation buffers, which seem to do what I want, but I've heard that they are very slow and anyway I can't tell whether they even exist in OpenGL ES 3, let alone how to use them.
I'm imagining something like the following (after setting up some kind of storage buffer):
- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect
{
glClear(...);
glBufferData(...);
glEnableVertexAttribArray(...);
glVertexAttribPointer(...);
// draw solid shape
glDrawArrays(GL_TRIANGLE_STRIP, ...);
// draw contents of storage buffer
// draw line
glDrawArrays(GL_LINE_STRIP, ...);
// multiply the alpha value of each pixel in the storage buffer by 0.9 to fade
// draw line again, this time into the storage buffer
// draw points
glDrawArrays(GL_POINTS, ...);
}
Is this possible? What are the commands I need to use (in particular, to combine the contents of the storage buffer and change its alpha)? And is this likely to actually be more efficient than the brute-force method?
I ended up achieving the desired result by rendering to a texture, as described for example here. The basic idea is to setup a custom framebuffer and attach a texture to it – I then render the line that I want to persist into this framebuffer (without clearing it) and render the whole framebuffer as a texture into the default framebuffer (which is cleared every frame). Instead of clearing the custom framebuffer, I render a slightly opaque quad over the whole screen to make the previous contents fade out a little every frame.
The relevant code is below; setting up the framebuffer and persistence texture is done in the init method:
// vertex data for fullscreen textured quad (x, y, texX, texY)
GLfloat persistVertexData[16] = {-1.0, -1.0, 0.0, 0.0,
-1.0, 1.0, 0.0, 1.0,
1.0, -1.0, 1.0, 0.0,
1.0, 1.0, 1.0, 1.0};
// setup texture vertex buffer
glGenBuffers(1, &persistVertexBuffer);
glBindBuffer(GL_ARRAY_BUFFER, persistVertexBuffer);
glBufferData(GL_ARRAY_BUFFER, sizeof(persistVertexData), persistVertexData, GL_STATIC_DRAW);
// create texture for persistence data and bind
glGenTextures(1, &persistTexture);
glBindTexture(GL_TEXTURE_2D, persistTexture);
// provide an empty image
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 2048, 1536, 0, GL_RGBA, GL_UNSIGNED_BYTE, 0);
// set texture parameters
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
// create frame buffer for persistence data
glGenFramebuffers(1, &persistFrameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, persistFrameBuffer);
// attach render buffer
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, persistTexture, 0);
// check for errors
NSAssert(glCheckFramebufferStatus(GL_FRAMEBUFFER) == GL_FRAMEBUFFER_COMPLETE, #"Error: persistence framebuffer incomplete!");
// initialize default frame buffer pointer
defaultFrameBuffer = -1;
and in the glkView:drawInRect: method:
// get default frame buffer id
if (defaultFrameBuffer == -1)
glGetIntegerv(GL_FRAMEBUFFER_BINDING, &defaultFrameBuffer);
// clear screen
glClear(GL_COLOR_BUFFER_BIT);
// DRAW PERSISTENCE
// bind persistence framebuffer
glBindFramebuffer(GL_FRAMEBUFFER, persistFrameBuffer);
// render full screen quad to fade
glEnableVertexAttribArray(...);
glBindBuffer(GL_ARRAY_BUFFER, persistVertexBuffer);
glVertexAttribPointer(...);
glUniform4f(colorU, 0.0, 0.0, 0.0, 0.01);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
// add most recent line
glBindBuffer(GL_ARRAY_BUFFER, dataVertexBuffer);
glVertexAttribPointer(...);
glUniform4f(colorU, color[0], color[1], color[2], 0.8*color[3]);
glDrawArrays(...);
// return to normal framebuffer
glBindFramebuffer(GL_FRAMEBUFFER, defaultFrameBuffer);
// switch to texture shader
glUseProgram(textureProgram);
// bind texture
glBindTexture(GL_TEXTURE_2D, persistTexture);
glUniform1i(textureTextureU, 0);
// set texture vertex attributes
glBindBuffer(GL_ARRAY_BUFFER, persistVertexBuffer);
glEnableVertexAttribArray(texturePositionA);
glEnableVertexAttribArray(textureTexCoordA);
glVertexAttribPointer(self.shaderBridge.texturePositionA, 2, GL_FLOAT, GL_FALSE, 4*sizeof(GLfloat), 0);
glVertexAttribPointer(self.shaderBridge.textureTexCoordA, 2, GL_FLOAT, GL_FALSE, 4*sizeof(GLfloat), 2*sizeof(GLfloat));
// draw fullscreen quad with texture
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
// DRAW NORMAL FRAME
glUseProgram(normalProgram);
glEnableVertexAttribArray(...);
glVertexAttribPointer(...);
// draw solid shape
glDrawArrays(GL_TRIANGLE_STRIP, ...);
// draw line
glDrawArrays(GL_LINE_STRIP, ...);
// draw points
glDrawArrays(GL_POINTS, ...);
The texture shaders are very simple: the vertex shader just passes the texture coordinate to the fragment shader:
attribute vec4 aPosition;
attribute vec2 aTexCoord;
varying vec2 vTexCoord;
void main(void)
{
gl_Position = aPosition;
vTexCoord = aTexCoord;
}
and the fragment shader reads the fragment color from the texture:
uniform highp sampler2D uTexture;
varying vec2 vTexCoord;
void main(void)
{
gl_FragColor = texture2D(uTexture, vTexCoord);
}
Although this works, it doesn't seem very efficient, causing the renderer utilization to rise to close to 100%. It only seems better than the brute force approach when the number of lines drawn each frame exceeds 100 or so. If anyone has any suggestions on how to improve this code, I would be very grateful!
I am adding WebGL support in my game, but I have a strange problem : it runs even slower than in Canvas 2D rendering mode, and I do not understand why.
I checked on both Firefox PC, Chrome PC, and Chrome Android, they run WebGL demos on the web with hundreds of sprites smoothly though, so I definitly made an error in my code.
Firefox's profiler says the whole game uses only 7% of the ressources, the rendering part takes only 1.2%. It is just the title screen of the game and there are only five sprites to draw. It is slow though...
update : Chrome's profiler says idle is only 4%, program is a huge 93%, and render 2.6%.
When using Canvas 2D things are very different, 76% idle, 16% program, 2.3% for the drawing function.
There definitly is a problem in my WebGL rendering code.
update : Android Chrome's profiler (on JXD S5110) always says program is ~39%, drawArrays is ~ 8%, bufferData ~5%, and bindTexture is 3%. Everything else is quite negligible.
If a function of mines was wasting all the ressources I would know what to do, but here the bottlenecks seem to be "program" (the browser itself ?) and webgl methods, two things I can't edit.
Please someone have a look at my code and tell me what I did wrong.
Here are my shaders
<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;
attribute vec2 a_texCoord;
uniform vec2 u_resolution;
uniform vec2 u_translation;
uniform vec2 u_rotation;
varying vec2 v_texCoord;
void main()
{
// Rotate the position
vec2 rotatedPosition = vec2(
a_position.x * u_rotation.y + a_position.y * u_rotation.x,
a_position.y * u_rotation.y - a_position.x * u_rotation.x);
// Add in the translation.
vec2 position = rotatedPosition + u_translation;
// convert the rectangle from pixels to 0.0 to 1.0
vec2 zeroToOne = a_position / u_resolution;
// convert from 0->1 to 0->2
vec2 zeroToTwo = zeroToOne * 2.0;
// convert from 0->2 to -1->+1 (clipspace)
vec2 clipSpace = zeroToTwo - 1.0;
gl_Position = vec4(clipSpace * vec2(1, -1), 0, 1);
// pass the texCoord to the fragment shader
// The GPU will interpolate this value between points
v_texCoord = a_texCoord;
}
</script>
<script id="2d-fragment-shader" type="x-shader/x-fragment">
precision mediump float;
// our texture
uniform sampler2D u_image;
// the texCoords passed in from the vertex shader.
varying vec2 v_texCoord;
void main()
{
// Look up a color from the texture.
gl_FragColor = texture2D(u_image, vec2(v_texCoord.s, v_texCoord.t));
}
</script>
Here is the creation code of my canvas and their contexts when in WebGL mode.
(I use to use several layered canvas in order to avoid drawing the backgrounds and foregrounds at every frame while they never change, that is why canvas and contexts are in arrays.)
// Get A WebGL context
liste_canvas[c] = document.createElement("canvas") ;
document.getElementById('game_div').appendChild(liste_canvas[c]);
liste_ctx[c] = liste_canvas[c].getContext('webgl',{premultipliedAlpha:false}) || liste_canvas[c].getContext('experimental-webgl',{premultipliedAlpha:false});
var gl = liste_ctx[c] ;
gl.viewport(0, 0, game.res_w, game.res_h);
// setup a GLSL program
gl.vertexShader = createShaderFromScriptElement(gl, "2d-vertex-shader");
gl.fragmentShader = createShaderFromScriptElement(gl, "2d-fragment-shader");
gl.program = createProgram(gl, [gl.vertexShader, gl.fragmentShader]);
gl.useProgram(gl.program);
// look up where the vertex data needs to go.
positionLocation = gl.getAttribLocation(gl.program, "a_position");
texCoordLocation = gl.getAttribLocation(gl.program, "a_texCoord");
// provide texture coordinates for the rectangle.
texCoordBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, texCoordBuffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
0.0, 0.0,
1.0, 0.0,
0.0, 1.0,
0.0, 1.0,
1.0, 0.0,
1.0, 1.0]), gl.STATIC_DRAW);
gl.enableVertexAttribArray(texCoordLocation);
gl.vertexAttribPointer(texCoordLocation, 2, gl.FLOAT, false, 0, 0);
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA);
gl.enable( gl.BLEND ) ;
gl.posBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, gl.posBuffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
0.0, 0.0,
1.0, 0.0,
0.0, 1.0,
0.0, 1.0,
1.0, 0.0,
1.0, 1.0]), gl.STATIC_DRAW);
gl.enableVertexAttribArray(positionLocation);
gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);
In the .onload function of my images, I add
var gl = liste_ctx[c] ;
this.texture = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, this.texture);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, this );
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.bindTexture(gl.TEXTURE_2D, null);
And here is the WebGL part of my draw_sprite() function :
var gl = liste_ctx[c] ;
gl.bindTexture(gl.TEXTURE_2D, sprites[d_sprite].texture);
var resolutionLocation = gl.getUniformLocation(gl.program, "u_resolution");
gl.uniform2f(resolutionLocation, liste_canvas[c].width, liste_canvas[c].height);
gl.bindBuffer(gl.ARRAY_BUFFER, gl.posBuffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
topleft_x , topleft_y ,
topright_x , topright_y ,
bottomleft_x , bottomleft_y ,
bottomleft_x , bottomleft_y ,
topright_x , topright_y ,
bottomright_x , bottomright_y ]), gl.STATIC_DRAW);
gl.drawArrays(gl.TRIANGLES, 0, 6);
What did I do wrong ?
This may help: What do the "Not optimized" warnings in the Chrome Profiler mean?
Relevant links:
https://groups.google.com/forum/#!topic/v8-users/_oZ4fUSitRY
https://github.com/petkaantonov/bluebird/wiki/Optimization-killers
For "optimized too many times", that means the function parameters / behavior change too much, so Chrome keeps having to re-optimize it.
What was making it so slow was using several webgl canvas, I use only one now and it works way better. But it is still a bit slower than Canvas 2D though, and the profiler says 65% is idle while it lags as hell so I really don't understand...
edit : I think I got it. Since my computer is running WinXP, hardware acceleration for WebGL can't be enabled, so the browsers use software rendering, and that explains why 'program' is huge in Chrome's profiler. However, hardware acceleration seems to work for 2d context, that is why it is faster.