Edge/outline detection from texture in fragment shader - webgl

I am trying to display sharp contours from a texture in WebGL.
I pass a texture to my fragment shaders then I use local derivatives to display the contours/outline, however, it is not smooth as I would expect it to.
Just printing the texture without processing works as expected:
vec2 texc = vec2(((vProjectedCoords.x / vProjectedCoords.w) + 1.0 ) / 2.0,
((vProjectedCoords.y / vProjectedCoords.w) + 1.0 ) / 2.0 );
vec4 color = texture2D(uTextureFilled, texc);
gl_FragColor = color;
With local derivatives, it misses some edges:
vec2 texc = vec2(((vProjectedCoords.x / vProjectedCoords.w) + 1.0 ) / 2.0,
((vProjectedCoords.y / vProjectedCoords.w) + 1.0 ) / 2.0 );
vec4 color = texture2D(uTextureFilled, texc);
float maxColor = length(color.rgb);
gl_FragColor.r = abs(dFdx(maxColor));
gl_FragColor.g = abs(dFdy(maxColor));
gl_FragColor.a = 1.;

In theory, your code is right.
But in practice most GPUs are computing derivatives on blocks of 2x2 pixels.
So for all 4 pixels of such block the dFdX and dFdY values will be the same.
(detailed explanation here)
This will cause some kind of aliasing and you will miss some pixels for the contour of the shape randomly (this happens when the transition from black to the shape color occurs at the border of a 2x2 block).
To fix this, and get the real per pixel derivative, you can instead compute it yourself, this would look like this :
// get tex coordinates
vec2 texc = vec2(((vProjectedCoords.x / vProjectedCoords.w) + 1.0 ) / 2.0,
((vProjectedCoords.y / vProjectedCoords.w) + 1.0 ) / 2.0 );
// compute the U & V step needed to read neighbor pixels
// for that you need to pass the texture dimensions to the shader,
// so let's say those are texWidth and texHeight
float step_u = 1.0 / texWidth;
float step_v = 1.0 / texHeight;
// read current pixel
vec4 centerPixel = texture2D(uTextureFilled, texc);
// read nearest right pixel & nearest bottom pixel
vec4 rightPixel = texture2D(uTextureFilled, texc + vec2(step_u, 0.0));
vec4 bottomPixel = texture2D(uTextureFilled, texc + vec2(0.0, step_v));
// now manually compute the derivatives
float _dFdX = length(rightPixel - centerPixel) / step_u;
float _dFdY = length(bottomPixel - centerPixel) / step_v;
// display
gl_FragColor.r = _dFdX;
gl_FragColor.g = _dFdY;
gl_FragColor.a = 1.;
A few important things :
texture should not use mipmaps
texture min & mag filtering should be set to GL_NEAREST
texture clamp mode should be set to clamp (not repeat)
And here is a ShaderToy sample, demonstrating this :

Related

WebGL is there a way to load dynamic buffers in fragment shaders?

I have a fragment shader that can draw an arc based on a set of parameters. The idea was to make the shader resolution independent, so I pass the center of the arc and the bounding radii as pixel values on the screen. You can then just render the shader by setting your vertex positions in the shape of a square. This is the shader:
precision mediump float;
#define PI 3.14159265359
#define _2_PI 6.28318530718
#define PI_2 1.57079632679
// inputs
vec2 center = u_resolution / 2.;
vec2 R = vec2( 100., 80. );
float ang1 = 1.0 * PI;
float ang2 = 0.8 * PI;
vec3 color = vec3( 0., 1.0, 0. );
// prog vars
uniform vec2 u_resolution;
float smOOth = 1.3;
vec3 bkgd = vec3( 0.0 ); // will be a sampler
void main () {
// get the dist from the current pixel to the coord.
float r = distance( gl_FragCoord.xy, center );
if ( r < R.x && r > R.y ) {
// If we are in the radius, do some trig to find the angle and normalize
// to
float theta = -( atan( gl_FragCoord.y - center.y,
center.x - gl_FragCoord.x ) ) + PI;
// This is to make sure the angles are clipped at 2 pi, but if you pass
// the values already clipped, then you can safely delete this and make
// the code more efficinent.
ang1 = mod( ang1, _2_PI );
ang2 = mod( ang2, _2_PI );
float angSum = ang1 + ang2;
bool thetaCond;
vec2 thBound; // short for theta bounds: used to calculate smoothing
// at the edges of the circle.
if ( angSum > _2_PI ) {
thBound = vec2( ang2, angSum - _2_PI );
thetaCond = ( theta > ang2 && theta < _2_PI ) ||
( theta < thetaBounds.y );
} else {
thBound = vec2( ang2, angSum );
thetaCond = theta > ang2 && theta < angSum;
}
if ( thetaCond ) {
float angOpMult = 10000. / ( R.x - R.y ) / smOOth;
float opacity = smoothstep( 0.0, 1.0, ( R.x - r ) / smOOth ) -
smoothstep( 1.0, 0.0, ( r - R.y ) / smOOth ) -
smoothstep( 1.0, 0.0, ( theta - thBound.x )
* angOpMult ) -
smoothstep( 1.0, 0.0, ( thBound.y - theta )
* angOpMult );
gl_FragColor = vec4( mix( bkgd, color, opacity ), 1.0 );
} else
discard;
} else
discard;
}
I figured this way of drawing a circle would yield better quality circles and be less hassle than loading a bunch of vertices and drawing triangle fans, even though it probably isn't as efficient. This works fine, but I don't just want to draw one fixed circle. I want to draw any circle I would want on the screen. So I had an idea to set the 'inputs' to varyings and pass a buffer with parameters to each of the vertices of a given bounding square. So my vertex shader looks like this:
attribute vec2 a_square;
attribute vec2 a_center;
attribute vec2 a_R;
attribute float a_ang1;
attribute float a_ang2;
attribute vec3 a_color;
varying vec2 center;
varying vec2 R;
varying float ang1;
varying float ang2;
varying vec3 color;
void main () {
gl_Position = vec4( a_square, 0.0, 1.0 );
center = a_center;
R = a_R;
ang1 = a_ang1;
ang2 = a_ang2;
color = a_color;
}
'a_square' is just the vertex for the bounding square that the circle would sit in.
Next, I define a buffer for the inputs for one test circle (in JS). One of the problems with doing it this way is that the circle parameters have to be repeated for each vertex, and for a box, this means four times. 'pw' and 'ph' are the width and height of the canvas, respectively.
var circleData = new Float32Array( [
pw / 2, ph / 2,
440, 280,
Math.PI * 1.2, Math.PI * 0.2,
1000, 0, 0,
pw/2,ph/2,440,280,Math.PI*1.2,Math.PI*0.2,1000,0,0,
pw/2,ph/2,440,280,Math.PI*1.2,Math.PI*0.2,1000,0,0,
pw/2,ph/2,440,280,Math.PI*1.2,Math.PI*0.2,1000,0,0,
] );
Then I simply load my data into a gl buffer (circleBuffer) and bind the appropriate attributes to it.
gl.bindBuffer( gl.ARRAY_BUFFER, bkgd.circleBuffer );
gl.vertexAttribPointer( bkgd.aCenter, 2, gl.FLOAT, false, 0 * floatSiz, 9 * floatSiz );
gl.enableVertexAttribArray( bkgd.aCenter );
gl.vertexAttribPointer( bkgd.aR, 2, gl.FLOAT, false, 2 * floatSiz, 9 * floatSiz );
gl.enableVertexAttribArray( bkgd.aR );
gl.vertexAttribPointer( bkgd.aAng1, 1, gl.FLOAT, false, 4 * floatSiz, 9 * floatSiz );
gl.enableVertexAttribArray( bkgd.aAng1 );
gl.vertexAttribPointer( bkgd.aAng2, 1, gl.FLOAT, false, 5 * floatSiz, 9 * floatSiz );
gl.enableVertexAttribArray( bkgd.aAng2 );
gl.vertexAttribPointer( bkgd.aColor, 3, gl.FLOAT, false, 6 * floatSiz, 9 * floatSiz );
gl.enableVertexAttribArray( bkgd.aColor );
When I load my page, I do see a circle, but it seems to me that the radii are the only attributes that are actually reflecting any type of responsiveness. The angles, center, and color are not reflecting the values they are supposed to be, and I have absolutely no idea why the radii are the only things that are actually working.
Nonetheless, this seems to be an inefficient way to load arguments into a fragment shader to draw a circle, as I have to reload the values for every vertex of the box, and then the GPU interpolates those values for no reason. Is there a better way to pass something like an attribute buffer to a fragment shader, or in general to use a fragment shader in this way? Or should I just use vertices to draw my circle instead?
If you're only drawing circles you can use instanced drawing to not repeat the info.
See this Q&A: what does instancing do in webgl
Or this article
Instancing lets you use some data per instance, as in per circle.
You can also use a texture to store the per circle data or all data. See this Q&A: How to do batching without UBOs?
Whether either are more or less efficient depends on the GPU/driver/OS/Browser. If you need to draw 1000s of circles this might be efficient. Most apps draw a variety of things so would chose a more generic solution unless they had special needs to draw 1000s of circles.
Also it may not be efficient because you're still calling the fragment shader for every pixel that is in the square but not in the circle. That's 30% more calls to the fragment shader than using triangles and that assumes your code is drawing quads that fit the circles. It looks at a glance that your actual code is drawing full canvas quads which is terribly inefficient.

draw a ellipse curve in fragment shader

I want an smooth curve in webgl so I tried to draw a rectangle in vertex shader and send positions to fragment shader in normalize size and Size parameter which is the real size of square
in vec2 Position;
in vec2 Size;
out vec2 vPosition;
out vec2 vSize;
void main() {
vSize = Size;
vPosition = Position
gl_Position = Position * Size
}
when size = [width, height] of square and is equal for every vertex and position = [
-1 , -1 ,
-1 , 1 ,
1 , -1 ,
1 , 1 ,
]
so my rectangle will be drawn in [2 * width , 2 * height] but I can do geometric operations in fragment shader with a 2 * 2 square which is normalized
but I have a problem with drawing ellipse(or circle with this sizes) in fragment shader when I want to make hollow circle with a thickness parameter it's thickness in horizontal direction is not same as vertical direction and I know it's because of that I'm using same size for horizontal and vertical directions(2,2) but in display they are different and this is the problem which make as you can see thickness in all of it is not same.
I want a geometry formula to calculate thickness in each direction then I can draw a hollow ellipse.
thanks in advance.
sorry for bad English
If you put heavy mathematical computing in your fragment shader, it will be slow.
The trick can be to use an approximation that can be visually acceptable.
Your problem is that the thickness is different on the vertical and the horizontal axis.
What you need is discarding fragments if the radius of the current point is greater than 1 and lower than radiusMin. Let uniWidth and uniHeight be the size of your rectangle.
* When y is null, on the horizontal axis, radiusMin = 1.0 - BORDER / uniWidth.
* When x is null, on the vertical axis, radiusMin = 1.0 - BORDER / uniHeight.
The trick is to interpolate between this two radii using the mix() function.
Take a look at my live example below to convince you that the result is not that bad.
Here is the fragment shader to do such a computation:
precision mediump float;
uniform float uniWidth;
uniform float uniHeight;
varying vec2 varCoords;
const float BORDER = 32.0;
void main() {
float x = varCoords.x;
float y = varCoords.y;
float radius = x * x + y * y;
if( radius > 1.0 ) discard;
radius = sqrt( radius );
float radiusH = 1.0 - BORDER / uniWidth;
float radiusV = 1.0 - BORDER / uniHeight;
float radiusAverage = (radiusH + radiusV) * 0.5;
float minRadius = 0.0;
x = abs( x );
y = abs( y );
if( x > y ) {
minRadius = mix( radiusH, radiusAverage, y / x );
}
else {
minRadius = mix( radiusV, radiusAverage, x / y );
}
if( radius < minRadius ) discard;
gl_FragColor = vec4(1, .5, 0, 1);
}
Here is a live example: https://jsfiddle.net/7rh2eog1/5/
There is a implicit formula for x and y which are in the blue area in hollow ellipse with thickness parameter, As we know the thickness and we have the size of our view we can make two ellipse with size1 = Size - vec2(thickness) and size2 = Size + vec2(thickness)
and then length(position/size1) < 1.0 < length(position/size2)

GPUImage Lookup Filter - creating a color depth greater than 512² colors

GPUImage's LookupFilter uses an RGB pixel map that's 512x512. When the filter executes, it creates a comparison between a modified version of this image with the original, and extrapolates an image filter.
The filter code is pretty straightforward. Here's an extract so you can see what's going on:
void main()
{
highp vec4 textureColor = texture2D(inputImageTexture, textureCoordinate);
highp float blueColor = textureColor.b * 63.0;
highp vec2 quad1;
quad1.y = floor(floor(blueColor) / 8.0);
quad1.x = floor(blueColor) - (quad1.y * 8.0);
highp vec2 quad2;
quad2.y = floor(ceil(blueColor) / 8.0);
quad2.x = ceil(blueColor) - (quad2.y * 8.0);
highp vec2 texPos1;
texPos1.x = (quad1.x * 0.125) + 0.5/512.0 + ((0.125 - 1.0/512.0) * textureColor.r);
texPos1.y = (quad1.y * 0.125) + 0.5/512.0 + ((0.125 - 1.0/512.0) * textureColor.g);
highp vec2 texPos2;
texPos2.x = (quad2.x * 0.125) + 0.5/512.0 + ((0.125 - 1.0/512.0) * textureColor.r);
texPos2.y = (quad2.y * 0.125) + 0.5/512.0 + ((0.125 - 1.0/512.0) * textureColor.g);
lowp vec4 newColor1 = texture2D(inputImageTexture2, texPos1);
lowp vec4 newColor2 = texture2D(inputImageTexture2, texPos2);
lowp vec4 newColor = mix(newColor1, newColor2, fract(blueColor));
gl_FragColor = mix(textureColor, vec4(newColor.rgb, textureColor.w), intensity);
}
);
See where the filter map is dependent on this being a 512x512 image?
I'm looking at ways to 4x the color depth here, using a 1024x1024 source image instead, but I'm not sure how this lookup filter image would have originally been generated.
Can something like this be generated in code? If so, I realize it's a very broad question, but how would I go about doing that? If it can't be generated in code, what are my options?
—-
Update:
Turns out the original LUT generation code was included in the header file all along. The questionable part here is from the header file:
Lookup texture is organised as 8x8 quads of 64x64 pixels representing all possible RGB colors:
How is 64x64 a map of all possible RGB channels? 64³ = 262,144 but that only accounts for 1/64th of the presumed 24-bit capacity of RGB, which is 64³ (16,777,216). What's going on here? Am I missing the way this LUT works? How are we accounting for all possible RGB colors with only 1/64th of the data?
for (int by = 0; by < 8; by++) {
for (int bx = 0; bx < 8; bx++) {
for (int g = 0; g < 64; g++) {
for (int r = 0; r < 64; r++) {
image.setPixel(r + bx * 64, g + by * 64, qRgb((int)(r * 255.0 / 63.0 + 0.5),
(int)(g * 255.0 / 63.0 + 0.5),
(int)((bx + by * 8.0) * 255.0 / 63.0 + 0.5)));
}
}
}
}
I'm not quite sure what problem you are actually having. When you say you want "4x the color depth" what do you actually mean. Color depth normally means the number of bits per color channel (or per pixel), which is totally independent of the resolution of the image.
In terms of lookup table accuracy (which is resolution dependent), assuming you are using bilinear filtered texture inputs from the original texture, and filtered lookups into the transform table, then you are already linearly interpolating between samples in the lookup table. Interpolation of color channels will be at higher precision than the storage format; e.g. often fp16 equivalent, even for textures stored at 8-bit per pixel.
Unless you have a significant amount of non-linearity in your color transform (not that common) adding more samples to the lookup table is unlikely to make a significant difference to the output - the interpolation will already be doing a reasonably good job of filling in the gaps.
Lev Zelensky provided the original work for this, so I'm not as familiar with how this works internally, but you can look at the math being performed in the shader to get an idea of what's going on.
In the 512x512 lookup, you have an 8x8 grid of cells. Within those cells, you have a 64x64 image patch. The red values go from 0 to 255 (0.0 to 1.0 in normalized values) going from left to right in that patch, and the green values go from 0 to 255 going down. That means that there are 64 steps in red, and 64 steps in green.
Each cell then appears to increase the blue value as you progress down the patches, left to right, top to bottom. With 64 patches, that gives you 64 blue values to match the 64 red and green ones. That gives you equal coverage across the RGB values in all channels.
So, if you wanted to double the number of color steps, you'd have to double the patch size to 128x128 and have 128 grids. It'd have to be more of a rectangle due to 128 not having an integer square root. Just going to 1024x1024 might let you double the color depth in the red and green channels, but blue would now be half their depth. Balancing the three out would be a little trickier than just doubling the image size.

Using shaders from Shadertoy in Interface Builder (Xcode)

I'm attempting to see what shaders look like in Interface Builder using sprite kit, and would like to use some of the shaders at ShaderToy. To do it, I created a "shader.fsh" file, a scene file, and added a color sprite to the scene, giving it a custom shader (shader.fsh)
While very basic shaders seem to work:
void main() {
gl_FragColor = vec4(0.0,1.0,0.0,1.0);
}
Any attempt I make to convert shaders from ShaderToy cause Xcode to freeze up (spinning color ball) as soon as the attempt is made to render them.
The shader I am working with for example, is this one:
#define M_PI 3.1415926535897932384626433832795
float rand(vec2 co)
{
return fract(sin(dot(co.xy ,vec2(12.9898,78.233))) * 43758.5453);
}
void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
float size = 30.0;
float prob = 0.95;
vec2 pos = floor(1.0 / size * fragCoord.xy);
float color = 0.0;
float starValue = rand(pos);
if (starValue > prob)
{
vec2 center = size * pos + vec2(size, size) * 0.5;
float t = 0.9 + 0.2 * sin(iGlobalTime + (starValue - prob) / (1.0 - prob) * 45.0);
color = 1.0 - distance(fragCoord.xy, center) / (0.5 * size);
color = color * t / (abs(fragCoord.y - center.y)) * t / (abs(fragCoord.x - center.x));
}
else if (rand(fragCoord.xy / iResolution.xy) > 0.996)
{
float r = rand(fragCoord.xy);
color = r * (0.25 * sin(iGlobalTime * (r * 5.0) + 720.0 * r) + 0.75);
}
fragColor = vec4(vec3(color), 1.0);
}
I've tried:
Replacing mainImage() with main(void) (so that it will be called)
Replacing the iXxxxx variables (iGlobalTime, iResolution) and fragCoord variables with their related variables (based on the suggestions here)
Replacing some of the variables (iGlobalTime)...
While changing mainImage to main() and swapping out the variables got it to work without error in TinyShading realtime tester app - the outcome is always the same in Xcode (spinning ball, freeze). Any advice here would be helpful as there is a surprisingly small amount of information currently available on the topic.
I managed to get this working in SpriteKit using SKShader. I've been able to render every shader from ShaderToy that I've attempted so far. The only exception is that you must remove any code using iMouse, since there is no mouse in iOS. I did the following...
1) Change the mainImage function declaration in the ShaderToy to...
void main(void) {
...
}
The ShaderToy mainImage function has an input named fragCoord. In iOS, this is globally available as gl_FragCoord, so your main function no longer needs any inputs.
2) Do a replace all to change the following from their ShaderToy names to their iOS names...
fragCoord becomes gl_FragCoord
fragColor becomes gl_FragColor
iGlobalTime becomes u_time
Note: There are more that I haven't encountered yet. I'll update as I do
3) Providing iResolution is slightly more involved...
iResolution is the viewport size (in pixels), which translates to the sprite size in SpriteKit. This used to be available as u_sprite_size in iOS, but has been removed. Luckily, Apple provides a nice example of how to inject it into your shader using uniforms in their SKShader documentation.
However, as stated in Shader Inputs section of ShaderToy, the type of iResolution is vec3 (x, y and z) as opposed to u_sprite_size, which is vec2 (x and y). I am yet to see a single ShaderToy that uses the z value of iResolution. So, we can simply use a z value of zero. I modified the example in the Apple documentation to provide my shader an iResolution of type vec3 like so...
let uniformBasedShader = SKShader(fileNamed: "YourShader.fsh")
let sprite = SKSpriteNode()
sprite.shader = uniformBasedShader
let spriteSize = vector_float3(
Float(sprite.frame.size.width), // x
Float(sprite.frame.size.height), // y
Float(0.0) // z - never used
)
uniformBasedShader.uniforms = [
SKUniform(name: "iResolution", vectorFloat3: spriteSize)
]
That's it :)
Here is the change to the shader that works when loaded as a shader with swift:
#define M_PI 3.1415926535897932384626433832795
float rand(vec2 co);
float rand(vec2 co)
{
return fract(sin(dot(co.xy ,vec2(12.9898,78.233))) * 43758.5453);
}
void main()
{
float size = 50.0; //Item 1:
float prob = 0.95; //Item 2:
vec2 pos = floor(1.0 / size * gl_FragCoord.xy);
float color = 0.0;
float starValue = rand(pos);
if (starValue > prob)
{
vec2 center = size * pos + vec2(size, size) * 0.5;
float t = 0.9 + 0.2 * sin(u_time + (starValue - prob) / (1.0 - prob) * 45.0); //Item 3:
color = 1.0 - distance(gl_FragCoord.xy, center) / (0.9 * size);
color = color * t / (abs(gl_FragCoord.y - center.y)) * t / (abs(gl_FragCoord.x - center.x));
}
else if (rand(v_tex_coord) > 0.996)
{
float r = rand(gl_FragCoord.xy);
color = r * (0.25 * sin(u_time * (r * 5.0) + 720.0 * r) + 0.75);
}
gl_FragColor = vec4(vec3(color), 1.0);
}
Play with Item 1: to increase the number of stars in the sky the smaller the number the more stars I like the number to be around 50 not too dense
Item 2: changes the randomness or how close together the stars will appear 1 = none, 0.1 = side by side around 0.75 gives a nice feel.
Item 3 is where most of the magic happens this is the size and pulse of the stars.
float t = 0.9
Changing 0.9, will increase the initial star sign up or down a nice value is 1.4 not too big not too small.
float t = 0.9 + 0.2
Changing the second value in this equation 0.2, will increase the pulse effect width of the stars proportionally to the original size I like with 1.4 a value of 1.2.
To add the shader to your swift project add a sprite to the scene the size of the screen then add the shader like this:
let backgroundImage = SKSpriteNode()
backgroundImage.texture = textureAtlas.textureNamed("any )
backgroundImage.size = screenSize
let shader = SKShader(fileNamed: "nightSky.fsh")
backgroundImage.shader = shader

iOs OpenGL ES 2.0 blending with shader on device and on simulator

Here is the fragment of fragment shader code (I need make blending in the shader):
float a = texture2D(tMask, texCoords).x; // opacity of current pixel from mask texture.
if ( a == 0.0)
discard;
else {
vec4 dst_clr = texture2D(tBkgText, posCoords); // color of current pixel in current framebuffer.
vec4 src_clr = vec4(vColor.rgb, sOpacity);
gl_FragColor = src_clr * a + dst_clr * (1.0 - a);
}
Here are the blending function and equation:
glBlendFunc(GL_ONE, GL_ZERO);
glBlendEquation(GL_FUNC_ADD);
And here are results on device (left) and on simulator (right):
How to make so that it works like on an emulator?
UPDATE:
I've removed discard:
float a = texture2D(tMask, texCoords).x; // opacity of current pixel from mask texture.
vec4 dst_clr = texture2D(tBkgText, posCoords); // color of current pixel in current framebuffer.
vec4 src_clr = vec4(vColor.rgb, sOpacity);
gl_FragColor = src_clr * a + dst_clr * (1.0 - a);
Now result on the device looks like:
I resolved this problem with adding glFlush after glDrawArrays. The problem was that the texture was not updated when the drawing occurs in its associated framebuffer.

Resources