I'm trying to create a fragment shader to recolor a 2D grayscale sprite but leave white and near-white fragments intact (ie: don't recolor pure white fragments, and only slightly recolor near-white fragments). I'm not sure how to do this without using a conditional branch which results in poor performance on certain hardware.
The existing shader in the game engine just performs a simple multiplication:
#ifdef GL_ES
precision lowp float;
#endif
varying vec4 v_fragmentColor;
varying vec2 v_texCoord;
uniform sampler2D CC_Texture0;
void main()
{
vec4 texColor = texture2D(CC_Texture0, v_texCoord);
gl_FragColor = texColor * v_fragmentColor;
}
I think that in order to avoid the conditional, I need some sort of continuous mathematical function that will recolor fragments with RGB values greater than, say, (0.9, 0.9, 0.9) less than it would for fragments which are less than (0.9, 0.9, 0.9).
Any help would be great!
I would do something like this: Calculate the fully-recolored pixel, then mix with the original based on a function. Here's an idea:
vec4 texColor = texture2D(CC_Texture0, v_texCoord);
const vec4 kLumWeights = vec4(.2126, .7152, .0722, 0.0); // Rec. 709 luminance weights
float luminance = dot (texColor, kLumWeights);
vec4 recolored = texColor * v_fragmentColor;
const float kThreshold = 0.8;
float mixAmount = (luminance - kThreshold) / (1.0 - kThreshold); // Everything below kThreshold becomes 0, and from kThreshold to 1.0 becomes 0 to 1.0
mixAmount = clamp (mixAmount, 0.0, 1.0);
gl_FragColor = mix (recolored, texColor, mixAmount);
Let me know if that works.
Related
I have this working scaled masked video texture over an image texture background. However it is positioned in the bottom left corner. I tried some tricks multiplying the coords but it doesn't seem to make much difference. I'll probably have to make alot of the values changeable uniforms but hardcoded ok for now.
What values can be used to change the video texture coords to display in the top right or bottom right corner ?
The video is a webcam stream with bodypix data providing the mask.
The alpha in mix is from bodypix data and needs to be calculated at 255 to properly display.
Fragment example
precision mediump float;
uniform sampler2D background;
uniform sampler2D frame;
uniform sampler2D mask;
uniform float texWidth;
uniform float texHeight;
void main(void) {
vec2 texCoord = gl_FragCoord.xy / vec2(texWidth,texHeight);
vec2 frameuv = texCoord * vec2(texWidth, texHeight) / vec2(200.0, 200.0);
vec4 texel0 = texture2D(background, texCoord);
vec4 frameTex = texture2D(frame, frameuv.xy);
vec4 maskTex = texture2D(mask, frameuv.xy);
gl_FragColor = mix(texel0, frameTex, step(frameuv.x, 1.0) * step(frameuv.y, 1.0) * maskTex.a * 255.);
}
https://jsfiddle.net/danrossi303/82tpoy94/3/
Instead of giving -1 to 1 values to my shaders, I would prefer giving them pixel values like for the 2D canvas context. So according to what I read, I did add a uniform variable which I set to the size of the canvas, and I divide.
But I must be missing something. The rendering is way too big...
gl_.resolutionLocation = gl.getUniformLocation( gl_.program , "u_resolution" );
gl.uniform4f(gl_.resolutionLocation , game.w , game.h , game.w , game.h );
My vertex shader :
attribute vec4 position;
attribute vec2 texcoord;
uniform vec4 u_resolution;
uniform mat4 u_matrix;
varying vec3 v_texcoord;
void main() {
vec4 zeroToOne = position / u_resolution ;
gl_Position = u_matrix * zeroToOne ;
v_texcoord = vec3(texcoord.xy, 1) * abs(position.x);
v_texcoord = v_texcoord/u_resolution.xyz ;
}
My fragment shader :
precision mediump float;
varying vec3 v_texcoord;
uniform sampler2D tex;
uniform float alpha;
void main()
{
gl_FragColor = texture2DProj(tex, v_texcoord);
gl_FragColor.rgb *= gl_FragColor.a ;
}
If you want to stay in pixels with code like the code you have then you'd want to apply the conversion to clip space after you've done everything in pixels.
In other words the code would be something like
rotatedPixelPosition = rotationMatrix * pixelPosition
clipSpacePosition = (rotatedPixelPosition / resolution) * 2.0 - 1.0;
So in other words you'd want
vec4 rotatedPosition = u_matrix * position;
vec2 zeroToOne = rotatedPosition.xy / u_resolution.xy;
vec2 zeroToTwo = zeroToOne * 2.0;
vec2 minusOneToPlusOne = zeroToTwo - 1.0;
vec2 clipspacePositiveYDown = minusOneToPlusOne * vec2(1, -1);
gl_Position = vec4(clipspacePositiveYDown, 0, 1);
If you do that and you set u_matrix to the identity then if position is in pixels you should see those positions at pixel positions. If u_matrix is strictly a rotation matrix the positions will rotate around the top left corner since rotation always happens around 0 and the conversion above puts 0 at the top left corner.
But really here's no reason to convert to from pixels to clip space by hand. You can instead convert and rotate all in the same matrix. This article covers that process. It starts with translate, rotation, scale, and converting from pixels to clip space with no matrices and converts it to something that does all of that combined using a single matrix.
Effectively
matrix = scaleYByMinusMatrix *
subtract1FromXYMatrix *
scaleXYBy2Matrix *
scaleXYBy1OverResolutionMatrix *
translationInPixelSpaceMatrix *
rotationInPixelSpaceMatrix *
scaleInPixelSpaceMatrix;
And then in your shader you only need
gl_Position = u_matrix * vec4(position, 0, 1);
Those top 4 matrixes are easy to compute as a single matrix, often called an orthographic projection in which case it simplifies to
matrix = projectionMatrix *
translationInPixelSpaceMatrix *
rotationInPixelSpaceMatrix *
scaleInPixelSpaceMatrix;
There's also this article which reproduces the matrix stack from canvas2D in WebGL
I am currently using this fragment shader in WebGL to apply highlights/shadows adjustments to photo textures.
The shader itself was pulled directly from the excellent GPUImage library for iOS.
uniform sampler2D inputImageTexture;
varying highp vec2 textureCoordinate;
uniform lowp float shadows;
uniform lowp float highlights;
const mediump vec3 luminanceWeighting = vec3(0.3, 0.3, 0.3);
void main()
{
lowp vec4 source = texture2D(inputImageTexture, textureCoordinate);
mediump float luminance = dot(source.rgb, luminanceWeighting);
mediump float shadow = clamp((pow(luminance, 1.0/(shadows+1.0)) + (-0.76)*pow(luminance, 2.0/(shadows+1.0))) - luminance, 0.0, 1.0);
mediump float highlight = clamp((1.0 - (pow(1.0-luminance, 1.0/(2.0-highlights)) + (-0.8)*pow(1.0-luminance, 2.0/(2.0-highlights)))) - luminance, -1.0, 0.0);
lowp vec3 result = vec3(0.0, 0.0, 0.0) + ((luminance + shadow + highlight) - 0.0) * ((source.rgb - vec3(0.0, 0.0, 0.0))/(luminance - 0.0));
gl_FragColor = vec4(result.rgb, source.a);
}
This shader as it stands, will only reduce highlights on a scale of 0.0 - 1.0. However I would like it to also brighten the highlights on a scale of 1.0-2.0.
With the aim of having a complete filter that reduces the images highlights when the highlights uniform is less than 1.0 and increases the intensity of the highlights when it is above 1.0. The same goes for the darkness shadows uniform
Highlights:
0.0(duller) ---- 1.0 (default - original pixel values) ----- 2.0 (brighter)
I have tried simply changing the clamp on the highlights variable to 0.0,2.0, and although this does indeed increase the brightness of the highlights when the uniform is above 1.0 it also seriously messes up the colors.
My understanding of image processing and constructing fragment shaders is extremely weak at best as you my be able to tell.
I'm just hoping someone can point me in the right direction.
EDIT:
Here are some example screenshots:-
The current filter with highlights set to 1.00 (basically the source image)
The current filter with highlights set to 0.00 as you can see the highlights get flattened/removed.
And finally here is what happens when I change the clamp in the fragment shader to allow values above 1.00 and set the highlights value to 2.00
I simply wish to be able to boost the highlights, making them brighter/more defined. i.e. the opposite of setting the value to 0.00
I don't really understand the shadow and highlight equations, but I can see that they are set up to never enhance shadows and highlights, but rather to wash them out. So we need a secondary step for enhancement.
For the highlights, I think to handle brighter colors, you need to blend towards white instead of adding something, so you don't get hue-shifts. I used a basic contrast equation to pick out the highlights, and then cubed it to clip out the midtones and shadows. The whiteTarget is just pulling out the top half of the 0.0-2.0 range to use as a multiplier to determine the strength of the brightening effect.
For the shadows, we are changing our range from 0.0-1.0 (where 0 is unchanged and 1 is washed out) to 0.0-2.0 (where 1 is unchanged and 2 is washed out). Therefore, the +1.0's in the shadow equation should be removed. Then for the 0.0-1.0 range, I just copied what I did for the highlights, except blending toward black. Maybe that can be optimized to avoid a mix function (not sure).
So here is my unoptimized version of the shader, set up so both shadows and highlights are on 0.0-2.0 scales, with 1.0 being the nominal. You might want to play around with those lines where I cube the luminance, and also with the value I used for contrast (currently 1.5), but it seems pretty good to me the way it is now--I adjusted it to try to avoid any ugly overlap between shadows and highlight ranges when the input parameters are at the two extremes.
uniform sampler2D inputImageTexture;
varying highp vec2 textureCoordinate;
uniform lowp float shadows;
uniform lowp float highlights;
const mediump vec3 luminanceWeighting = vec3(0.3, 0.3, 0.3);
void main()
{
lowp vec4 source = texture2D(inputImageTexture, textureCoordinate);
mediump float luminance = dot(source.rgb, luminanceWeighting);
//(shadows+1.0) changed to just shadows:
mediump float shadow = clamp((pow(luminance, 1.0/shadows) + (-0.76)*pow(luminance, 2.0/shadows)) - luminance, 0.0, 1.0);
mediump float highlight = clamp((1.0 - (pow(1.0-luminance, 1.0/(2.0-highlights)) + (-0.8)*pow(1.0-luminance, 2.0/(2.0-highlights)))) - luminance, -1.0, 0.0);
lowp vec3 result = vec3(0.0, 0.0, 0.0) + ((luminance + shadow + highlight) - 0.0) * ((source.rgb - vec3(0.0, 0.0, 0.0))/(luminance - 0.0));
// blend toward white if highlights is more than 1
mediump float contrastedLuminance = ((luminance - 0.5) * 1.5) + 0.5;
mediump float whiteInterp = contrastedLuminance*contrastedLuminance*contrastedLuminance;
mediump float whiteTarget = clamp(highlights, 1.0, 2.0) - 1.0;
result = mix(result, vec3(1.0), whiteInterp*whiteTarget);
// blend toward black if shadows is less than 1
mediump float invContrastedLuminance = 1.0 - contrastedLuminance;
mediump float blackInterp = invContrastedLuminance*invContrastedLuminance*invContrastedLuminance;
mediump float blackTarget = 1.0 - clamp(shadows, 0.0, 1.0);
result = mix(result, vec3(0.0), blackInterp*blackTarget);
gl_FragColor = vec4(result, source.a);
}
By the way, any idea why the original result line keeps adding 0's to everything? Seems like it could be simplified to
vec3 result = (luminance + shadow + highlight) * source.rgb / luminance;
But maybe it's a trick to cast to lowp within the calculation instead of after the calculation. Just a guess.
I have a problem with multiple drawing of textures in my program.
Blending mode is
glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glBlendEquation(GL_FUNC_ADD);
The value of a-channel is passed into the shader from cpu-code.
precision highp float;
uniform sampler2D tShape;
uniform vec4 vColor;
uniform float sOpacity;
varying vec4 texCoords;\n"
void main() {
float a = texture2D(tShape, texCoords.xy).x * sOpacity;
gl_FragColor = vec4(vColor.rgb, a);
}
It's calculated previously with
O = pow(O, 1.3);
for the best visual effect.
I draw with color (0; 0; 0) on the black transparent canvas (0;0;0;0), but with very low opacity:
0.03 -> 0.01048
0.06 -> 0.0258
0.09 -> 0.0437
0.12 -> 0.0635
...
I expect, that maximal value of point's color will be (0;0;0;1) (black, no transparent) after multiple drawings as on the simulator:
but it isn't so on the device:
Do you have any ideas, why is it so?
UPDATE:
Also manual blending works incorrect too (and with difference from standard).
glBlendFunc(GL_ONE, GL_ZERO);
Fragment shader code:
#extension GL_EXT_shader_framebuffer_fetch : require
precision highp float;
uniform sampler2D tShape;
uniform vec4 vColor;
uniform float sOpacity;
varying vec4 texCoords;
void main() {
float a = texture2D(tShape, texCoords.xy).x * sOpacity;
gl_FragColor = vec4(vColor.rgb * a, a) + (gl_LastFragData[0] * (1.0 - a));
}
Result on the simulator:
Result on the device:
I'm trying to understand your approach so I wrote down some equations:
This is how a new drawing is performed (If didn't make any mistake):
color = {pencil_shape}*sourceAlpha + {old_paint}*(1-sourceAlpha)
alpha = {pencil_shape} + {old_paint}*(1-sourceAlpha)
So basically you alpha is getting closer to 1 on each frame, and you color is blended each time based on the src alpha in the *pencil_shape*.
Questions:
Do you intend to use the alpha in the output image for anything?
Is your *pencil_shape* all black (0, 0, 0, 0)? (besides the cornes where I suppose it has some antialiasing effect)
After some experiments I've understood, that this problem is in supported precision of device . So on the iPad Air this problem appears less than on iPad 4, 3.
I have just completed the first version of my iOS app, Corebox, and am now working on some new features.
One of the new features is a "small" tweak to the OpenGL rendering to force some objects to never be drawn smaller than a minimum size. All of the objects needing this treatment are simple 2 point lines drawn with GL_LINES.
This annotated screenshot explains what I'm after. Ignore the grey lines, the only objects I'm interested in altering are the yellow wider lines.
I have googled this extensively and it seems what I need to do is alter the geometry of the lines using a vertex shader. I'm quite new to GLSL and most shader examples I can find deal with applying lighting and other effects, eg: GLSL Heroku Editor and KicksJS shader editor.
My current vertex shader is extremely basic:
// GL_LINES vertex shader
uniform mat4 Projection;
uniform mat4 Modelview;
attribute vec4 Position;
attribute vec4 SourceColor;
varying vec4 DestinationColor;
void main(void) {
DestinationColor = SourceColor;
gl_Position = Projection * Modelview * Position;
}
As is my fragment shader:
// GL_LINES fragment shader
varying lowp vec4 DestinationColor;
void main(void) {
gl_FragColor = DestinationColor;
}
My guess as to what is required:
Determine the distance between the viewer (camera position) and the object
Determine how big the object is on the screen, based on its size and distance from camera
If the object will be too small then adjust its vertices such that it becomes large enough to easily see on the screen.
Caveats and other notes:
But if you zoom out won't this cause the model to be just a blob of orange on the screen? Yes, this is exactly the effect I'm after.
Edit: Here is the final working version implementing suggestions by mifortin
uniform mat4 Projection;
uniform mat4 Modelview;
uniform float MinimumHeight;
attribute vec4 Position;
attribute vec4 ObjectCenter;
attribute vec4 SourceColor;
varying vec4 DestinationColor;
void main(void) {
// screen-space position of this vertex
vec4 screenPosition = Projection * Modelview * Position;
// screen-space mid-point of the object this vertex belongs to
vec4 screenObjectCenter = Projection * Modelview * ObjectCenter;
// Z should be 0 by this time and the projective transform in w.
// scale so w = 1 (these two should be in screen-space)
vec2 newScreenPosition = screenPosition.xy / screenPosition.w;
vec2 newObjectCenter = screenObjectCenter.xy / screenObjectCenter.w;
float d = distance(newScreenPosition, newObjectCenter);
if (d < MinimumHeight && d > 0.0) {
// Direction of this object, this really only makes sense in the context
// of a line (eg: GL_LINES)
vec2 towards = normalize(newScreenPosition - newObjectCenter);
// Shift the center point then adjust the vertex position accordingly
// Basically this converts: *--x--* into *--------x--------*
newObjectCenter = newObjectCenter + towards * MinimumHeight;
screenPosition.xy = newObjectCenter.xy * screenPosition.w;
}
gl_Position = screenPosition;
DestinationColor = SourceColor;
}
Note that I didn't test the code, but it should illustrate the solution.
If you want to use shaders, add in another uniform vec4 that is the center position of your line. Then you can do something similar to (note center could be precomputed on the CPU once):
uniform float MIN; //Minimum size of blob on-screen
uniform vec4 center; //Center of the line / blob
...
vec4 screenPos = Projection * Modelview * Position;
vec4 center = Projection * Modelview * Position;
//Z should be 0 by this time and the projective transform in w.
//scale so w = 1 (these two should be in screen-space)
vec2 nScreenPos = screenPos.xy / screenPos.w;
vec2 nCenter = center.xy / center.w;
float d = distance(nScreenPos, nCenter);
if (d < MIN && d > 0)
{
vec2 towards = normalize(nScreenPos - nCenter);
nCenter = nCenter + towards * MIN;
screenPos.xy = nCenter.xy * screenPos.w;
}
gl_Position = screenPos;
Find where on the screen the vertex would be drawn, then from the center of the blob stretch it if needed to ensure a minimum size.
This example is for round objects. For corners, you could make MIN an attribute so the distance from the center varies on a per-vertex basis.
If you just want something more box-like, check that the minimum distance of the x and y coordinates separately.
On the CPU, you could compute the coordinates in screen-space and scale accordingly before submitting to the GPU.