I have a code where I receive YUV channels and I'm drawing them using OpenGLES. Basically, I have a shader that combines them together.
I would like to add a sharpen filter to the result (using the following example: http://igortrindade.wordpress.com/2010/04/23/fun-with-opengl-and-shaders/)
I'm not sure how to run another shader on the actual result (since I would like to run it after my previous shader combined all channels to a single frame).
My current code looks like that:
glUniform1i(texLum, 0);
glUniform1i(texU, 1);
glUniform1i(texV, 2);
glEnableVertexAttribArray(positionLoc);
glVertexAttribPointer(positionLoc,
4,
GL_FLOAT,
GL_FALSE,
0,
&vertices[0]);
glEnableVertexAttribArray(texCoordLoc);
glVertexAttribPointer(texCoordLoc,
2,
GL_FLOAT,
GL_FALSE,
0,
&texCoords[0]);
glDrawElements(GL_TRIANGLES, sizeof(indices)/sizeof(indices[0]), GL_UNSIGNED_BYTE, &indices[0]);
I guess I need to add the new shader right before the last line (glDrawElements), but I'm not sure how to call it.
My shader looks like this:
static char const *frag =
"uniform lowp sampler2D texLum; \n"
"uniform lowp sampler2D texU; \n"
"uniform lowp sampler2D texV; \n"
"varying mediump vec2 texCoordAtFrag; \n"
"void main() { \n"
" lowp float Y = texture2D(texLum, texCoordAtFrag).r; \n"
" lowp float U = texture2D(texU, texCoordAtFrag).r; \n"
" lowp float V = texture2D(texV, texCoordAtFrag).r; \n"
" lowp float R = 1.164 * (Y - 16.0 / 256.0) + 1.596 * (V - 0.5); \n"
" lowp float G = 1.164 * (Y - 16.0 / 256.0) - 0.813 * (V - 0.5) - 0.391 * (U - 0.5); \n"
" lowp float B = 1.164 * (Y - 16.0 / 256.0) + 2.018 * (U - 0.5); \n"
" gl_FragColor = vec4(R,G,B,1); \n"
"}\r\n";
static char const *vert =
"varying mediump vec2 texCoordAtFrag; \n"
"attribute vec4 Position; \n"
"attribute vec2 TexCoord; \n"
"void main() { \n"
" texCoordAtFrag = TexCoord; \n"
" gl_Position = Position; \n"
"}\r\n";
Where texLum,texU,texV are the textures holding the channels.
Sharpen is a convolution filter, so it reads nine input values to produce one output value. So if you have another shader that is supposed to occur before it and operates one pixel at a time, there'd be a decent argument for running them in two steps (YUV transformation first, sharpen second) so as to eliminate repeated calculations, even if it weren't also the easiest way to combine shaders as closed boxes.
If you want to combine them live, break the YUV transformation into a separate function and have the sharpen filter call that function instead of texture2D. The GL shader compiler doesn't have any sort of restriction on the number of source files you can link together to make a compiled program precisely so that you can follow the usual programming routes for function reuse.
If you'd prefer to run one then run the other then use an intermediate render-to-texture stage. Do the YUV transform, then switch buffers and use the output of that as the input to the sharpen.
In practice the former may actually be faster than the latter since a YUV transform is probably a fast operation (if it's YUV to RGB then it's one matrix multiplication, for example) whereas memory speed and the need to do quite a drastic change state can be quite expensive. You'd probably need to profile if performance is a concern.
EDIT: so, from your current main you could just adapt that to (typed here, as I go, please forgive errors):
"uniform lowp sampler2D texLum; \n"
"uniform lowp sampler2D texU; \n"
"uniform lowp sampler2D texV; \n"
"varying mediump vec2 texCoordAtFrag; \n"
"lowp vec4 yuvTexture2D(mediump vec2 coord) { \n"
" lowp float Y = texture2D(texLum, coord).r; \n"
" lowp float U = texture2D(texU, coord).r; \n"
" lowp float V = texture2D(texV, coord).r; \n"
" lowp float R = 1.164 * (Y - 16.0 / 256.0) + 1.596 * (V - 0.5); \n"
" lowp float G = 1.164 * (Y - 16.0 / 256.0) - 0.813 * (V - 0.5) - 0.391 * (U - 0.5); \n"
" lowp float B = 1.164 * (Y - 16.0 / 256.0) + 2.018 * (U - 0.5); \n"
" return vec4(R,G,B,1.0); \n"
"}\r\n
And then in your sharpen filter you'd substitute the call to texture2D(<whatever>, coord) with a call to yuvTexture2D(coord), having either included the fragment in the source listing for the sharpen shader or linked it into the program. With respect to switching to using a matrix approach, I guess you'd want (I'm going to format it other than as a string constant, for ease of typing):
uniform lowp sampler2D texLum;
uniform lowp sampler2D texU;
uniform lowp sampler2D texV;
varying mediump vec2 texCoordAtFrag;
const mediump mat4 yuvToRgb =
mat4( 1.164, 1.164, 1.164, -0.07884,
2.018, -0.391, 0.0, 1.153216,
0.0, -0.813, 1.596, 0.53866,
0.0, 0.0, 0.0, 1.0);
lowp vec4 yuvTexture2D(mediump vec2 coord) {
lowp vec4 yuv =
vec4(
texture2D(texLum, coord).r,
texture2D(texU, coord).r,
texture2D(texV, coord).r,
1.0)
return yuvToRgb * yuv;
}
Related
How can I apply such a mask
to get effect such as bokeh
need to blur edge in mask and apply on image texture. How do that?
Vertex shader:
attribute vec4 a_Position;
void main()
{
gl_Position = a_Position;
}
Fragment shader:
precision lowp float;
uniform sampler2D u_Sampler; // textureSampler
uniform sampler2D u_Mask; // maskSampler
uniform vec3 iResolution;
vec4 blur(sampler2D source, vec2 size, vec2 uv) {
vec4 C = vec4(0.0);
float width = 1.0 / size.x;
float height = 1.0 / size.y;
float divisor = 0.0;
for (float x = -25.0; x <= 25.0; x++)
{
C += texture2D(source, uv + vec2(x * width, 0.0));
C += texture2D(source, uv + vec2(0.0, x * height));
divisor++;
}
C*=0.5;
return vec4(C.r / divisor, C.g / divisor, C.b / divisor, 1.0);
}
void main()
{
vec2 uv = gl_FragCoord.xy / iResolution.xy;
vec4 videoColor = texture2D(u_Sampler, uv);
vec4 maskColor = texture2D(u_Mask, uv);
gl_FragColor = blur(u_Sampler, iResolution.xy, uv);
}
vec4 blurColor = blur(u_Sampler, iResolution.xy, uv);
gl_FragColor = mix(blurColor, videoColor, maskColor.r);
But FYI it's not common to blur in one pass like you have. It's more common to blur in one direction (horizontally), then blur the result of that vertically, then mix the results, blurredTexture, videoTexture, mask.
In Brad Larson's excellent GPUImage, there is a halftone filter which also turns the picture black and white. I am just wanting the halftone effect without the black and white and I was wondering can anyone tell me how what I can remove from the following code to fix this? Have been playing around with it, but virtually have no experience in openGL and am not sure what to eliminate.
NSString *const kGPUImageHalftoneFragmentShaderString = SHADER_STRING
(
varying highp vec2 textureCoordinate;
uniform sampler2D inputImageTexture;
uniform highp float fractionalWidthOfPixel;
uniform highp float aspectRatio;
uniform highp float dotScaling;
const highp vec3 W = vec3(0.2125, 0.7154, 0.0721);
void main()
{
highp vec2 sampleDivisor = vec2(fractionalWidthOfPixel, fractionalWidthOfPixel / aspectRatio);
highp vec2 samplePos = textureCoordinate - mod(textureCoordinate, sampleDivisor) + 0.5 * sampleDivisor;
highp vec2 textureCoordinateToUse = vec2(textureCoordinate.x, (textureCoordinate.y * aspectRatio + 0.5 - 0.5 * aspectRatio));
highp vec2 adjustedSamplePos = vec2(samplePos.x, (samplePos.y * aspectRatio + 0.5 - 0.5 * aspectRatio));
highp float distanceFromSamplePoint = distance(adjustedSamplePos, textureCoordinateToUse);
lowp vec3 sampledColor = texture2D(inputImageTexture, samplePos ).rgb;
highp float dotScaling = 1.0 - dot(sampledColor, W);
lowp float checkForPresenceWithinDot = 1.0 - step(distanceFromSamplePoint, (fractionalWidthOfPixel * 0.5) * dotScaling);
gl_FragColor = vec4(vec3(checkForPresenceWithinDot), 1.0);
}
);
You can change the last line to
gl_FragColor = vec4(checkForPresenceWithinDot * sampledColor, 1.0);
This will make the effect have color instead of black and white only.
For the device, all of my shaders load fine except one. For this shader program I get "Fragment program failed to compile with current context state" error, followed by a similar error for the vertex shader when I make a call to glGetProgramInfoLog(...);
Vertex shader:
#version 100
uniform mat4 Projection;
uniform mat4 Modelview;
uniform mat4 Rotation;
uniform vec3 Translation;
uniform vec4 LightDirection;
uniform vec4 MaterialDiffuse;
uniform float MaterialShininess;
attribute vec3 position;
attribute vec3 normal;
varying vec4 color;
varying float specularCoefficient;
void main() {
vec3 _normal = normalize(mat3(Modelview[0].xyz, Modelview[1].xyz, Modelview[2].xyz)*normal);
// There is an easier way to do the above using typecast, but is apparently broken
float NdotL = dot(-_normal, normalize(vec3(LightDirection)));
if(NdotL < 0.0){
NdotL = 0.0;
}
color = NdotL * MaterialDiffuse;
float NdotO = dot(-_normal, vec3(0.0, 0.0, -1.0));
if(NdotO < 0.0){
NdotO = 0.0;
}
specularCoefficient = pow(NdotO, MaterialShininess);
vec3 p = position + Translation;
gl_Position = Projection*Modelview*vec4(p, 1.0);
}
Fragment shader:
#version 100
precision mediump float;
varying vec4 color;
varying float specularCoefficient;
uniform vec4 MaterialSpecular;
void main(){
gl_FragColor = vec4((color + specularCoefficient*MaterialSpecular).rgb, 1.0);
}
I am not sure what is going on, especially since I have a similar program that is exactly as above with the addition of texture coordinates. Also, I checked the compile status of each shader when I linked the programs using glGetShaderiv(theShader, GL_COMPILE_STATUS, &result) and they all checked out fine. Any ideas?
Changing the line
gl_FragColor = vec4((color + specularCoefficient*MaterialSpecular).rgb, 1.0);
in the fragment shader to
gl_FragColor = vec4((1.0*color + specularCoefficient*MaterialSpecular).rgb, 1.0);
fixes the problem. I suspect it has something to do with precision related to the varying variable color, for a reordering of the line to
gl_FragColor = vec4((MaterialSpecular + specularCoefficient*color).rgb, 1.0);
works as well.
I re-implemented Apple's GLImageProcessing with OpenGL ES 2 shaders. The effects are perfect but the performance of the Sharpness filter is not as good — it runs only at 20 FPS.
The shader code is simple:
Pass 0 for horizontal blur.
Pass 1 for vertical blur.
Pass 2 to mix the blur texture with the original texture.
Basically, the texture mix in Pass 2 is the cause of slowness since Pass 0 and Pass 1 are only done once and do not contribute to the bad performance.
How can I improve the performance?
Vertex shader:
attribute vec4 a_position;
attribute vec2 a_texCoord;
varying highp vec2 v_texCoord;
varying highp vec2 v_texCoord1;
varying highp vec2 v_texCoord2;
varying highp vec2 v_texCoord1_;
varying highp vec2 v_texCoord2_;
uniform mat4 u_modelViewProjectionMatrix;
uniform lowp int u_pass;
const highp float blurSizeH = 1.0 / 320.0;
const highp float blurSizeV = 1.0 / 480.0;
void main()
{
v_texCoord = a_texCoord;
if (u_pass == 0) {
v_texCoord1 = a_texCoord + vec2(1.3846153846 * blurSizeH, 0.0);
v_texCoord1_ = a_texCoord - vec2(1.3846153846 * blurSizeH, 0.0);
v_texCoord2 = a_texCoord + vec2(3.2307692308 * blurSizeH, 0.0);
v_texCoord2_ = a_texCoord - vec2(3.2307692308 * blurSizeH, 0.0);
} else if (u_pass == 1) {
v_texCoord1 = a_texCoord + vec2(0.0, 1.3846153846 * blurSizeV);
v_texCoord1_ = a_texCoord - vec2(0.0, 1.3846153846 * blurSizeV);
v_texCoord2 = a_texCoord + vec2(0.0, 3.2307692308 * blurSizeV);
v_texCoord2_ = a_texCoord - vec2(0.0, 3.2307692308 * blurSizeV);
}
gl_Position = u_modelViewProjectionMatrix * a_position;
}
Fragment shader:
varying highp vec2 v_texCoord;
varying highp vec2 v_texCoord1;
varying highp vec2 v_texCoord2;
varying highp vec2 v_texCoord1_;
varying highp vec2 v_texCoord2_;
uniform lowp int u_pass;
uniform sampler2D u_texture;
uniform sampler2D u_degenTexture;
uniform mediump mat4 u_filterMat;
void main()
{
if (u_pass == 0) {
gl_FragColor = texture2D(u_texture, v_texCoord) * 0.2270270270;
gl_FragColor += texture2D(u_texture, v_texCoord1) * 0.3162162162;
gl_FragColor += texture2D(u_texture, v_texCoord1_) * 0.3162162162;
gl_FragColor += texture2D(u_texture, v_texCoord2) * 0.0702702703;
gl_FragColor += texture2D(u_texture, v_texCoord2_) * 0.0702702703;
} else if (u_pass == 1) {
gl_FragColor = texture2D(u_degenTexture, v_texCoord) * 0.2270270270;
gl_FragColor += texture2D(u_degenTexture, v_texCoord1) * 0.3162162162;
gl_FragColor += texture2D(u_degenTexture, v_texCoord1_) * 0.3162162162;
gl_FragColor += texture2D(u_degenTexture, v_texCoord2) * 0.0702702703;
gl_FragColor += texture2D(u_degenTexture, v_texCoord2_) * 0.0702702703;
} else {
gl_FragColor = u_filterMat * texture2D(u_texture, v_texCoord) + (mat4(1.0) - u_filterMat) * texture2D(u_degenTexture, v_texCoord);
}
}
Before you continue with this, may I suggest you take a look at my open source GPUImage project? I have several hand-optimized sharpening effects in there, including the unsharp mask you're attempting here. I also make it reasonably easy to pull in image and video sources.
To your specific question, there are a couple of reasons why your shader is running slower than expected. The first is that you are using branching within your fragment shader. This kills performance on iOS devices, and should be avoided if at all possible. If you really need to have different conditions for different passes, split these apart into separate shader programs and swap the programs out as needed rather than using a uniform for control flow.
I'm also not sure that writing to gl_FragColor repeatedly is the fastest thing you can do here. I'd use a lowp or mediump intermediate color variable, add your Gaussian components to that, and then write the final result to gl_FragColor when done.
I do see that you've moved your sampling offset calculations to the vertex shader, and then passed those offsets into the fragment shader, which is a good thing that people usually miss. Once you implement the above tweaks (or give my framework a try to see how I handle this), you should get much better results from your filtering.
It turns out that it is really simple. The cause of bad performance is matrix-vector multiplication:
varying highp vec2 v_texCoord;
uniform sampler2D u_texture;
uniform sampler2D u_degenTexture;
uniform lowp float u_filterValue;
void main()
{
gl_FragColor = u_filterMat * texture2D(u_texture, v_texCoord) + (mat4(1.0) - u_filterMat) * texture2D(u_degenTexture, v_texCoord);
}
I initially wrote my code using matrix this way so that all my filters could share the same color mixing code. Now that I learned the lesson, I simply go back to write filter specific code and use scalar operations as much as possible:
varying highp vec2 v_texCoord;
uniform sampler2D u_texture;
uniform sampler2D u_degenTexture;
uniform lowp float u_filterValue;
void main()
{
gl_FragColor = u_filterValue * texture2D(u_texture, v_texCoord) + (1.0 - u_filterValue) * texture2D(u_degenTexture, v_texCoord);
}
Now it is awesome 60 fps!
Never thought it was such a naive issue, but it is.
Is there a way to get a Motion Blur effect on a UIImage?
I tried GPUImage, Filtrr and the iOS Core Image but all of these have regular blur - no motion blur.
I also tried UIImage-DSP but it's Motion Blur is almost non visible. I need something much stronger.
As I commented on the repository, I just added motion and zoom blurs to GPUImage. These are the GPUImageMotionBlurFilter and GPUImageZoomBlurFilter classes. This is an example of the zoom blur:
For the motion blur, I do a 9-hit Gaussian blur over a single direction. This is achieved using the following vertex and fragment shaders:
Vertex:
attribute vec4 position;
attribute vec4 inputTextureCoordinate;
uniform highp vec2 directionalTexelStep;
varying vec2 textureCoordinate;
varying vec2 oneStepBackTextureCoordinate;
varying vec2 twoStepsBackTextureCoordinate;
varying vec2 threeStepsBackTextureCoordinate;
varying vec2 fourStepsBackTextureCoordinate;
varying vec2 oneStepForwardTextureCoordinate;
varying vec2 twoStepsForwardTextureCoordinate;
varying vec2 threeStepsForwardTextureCoordinate;
varying vec2 fourStepsForwardTextureCoordinate;
void main()
{
gl_Position = position;
textureCoordinate = inputTextureCoordinate.xy;
oneStepBackTextureCoordinate = inputTextureCoordinate.xy - directionalTexelStep;
twoStepsBackTextureCoordinate = inputTextureCoordinate.xy - 2.0 * directionalTexelStep;
threeStepsBackTextureCoordinate = inputTextureCoordinate.xy - 3.0 * directionalTexelStep;
fourStepsBackTextureCoordinate = inputTextureCoordinate.xy - 4.0 * directionalTexelStep;
oneStepForwardTextureCoordinate = inputTextureCoordinate.xy + directionalTexelStep;
twoStepsForwardTextureCoordinate = inputTextureCoordinate.xy + 2.0 * directionalTexelStep;
threeStepsForwardTextureCoordinate = inputTextureCoordinate.xy + 3.0 * directionalTexelStep;
fourStepsForwardTextureCoordinate = inputTextureCoordinate.xy + 4.0 * directionalTexelStep;
}
Fragment:
precision highp float;
uniform sampler2D inputImageTexture;
varying vec2 textureCoordinate;
varying vec2 oneStepBackTextureCoordinate;
varying vec2 twoStepsBackTextureCoordinate;
varying vec2 threeStepsBackTextureCoordinate;
varying vec2 fourStepsBackTextureCoordinate;
varying vec2 oneStepForwardTextureCoordinate;
varying vec2 twoStepsForwardTextureCoordinate;
varying vec2 threeStepsForwardTextureCoordinate;
varying vec2 fourStepsForwardTextureCoordinate;
void main()
{
lowp vec4 fragmentColor = texture2D(inputImageTexture, textureCoordinate) * 0.18;
fragmentColor += texture2D(inputImageTexture, oneStepBackTextureCoordinate) * 0.15;
fragmentColor += texture2D(inputImageTexture, twoStepsBackTextureCoordinate) * 0.12;
fragmentColor += texture2D(inputImageTexture, threeStepsBackTextureCoordinate) * 0.09;
fragmentColor += texture2D(inputImageTexture, fourStepsBackTextureCoordinate) * 0.05;
fragmentColor += texture2D(inputImageTexture, oneStepForwardTextureCoordinate) * 0.15;
fragmentColor += texture2D(inputImageTexture, twoStepsForwardTextureCoordinate) * 0.12;
fragmentColor += texture2D(inputImageTexture, threeStepsForwardTextureCoordinate) * 0.09;
fragmentColor += texture2D(inputImageTexture, fourStepsForwardTextureCoordinate) * 0.05;
gl_FragColor = fragmentColor;
}
As an optimization, I calculate the step size between texture samples outside of the fragment shader by using the angle, blur size, and the image dimensions. This is then passed into the vertex shader, so that I can calculate the texture sampling positions there and interpolate across them in the fragment shader. This avoids dependent texture reads on iOS devices.
The zoom blur is much slower, because I still do these calculations in the fragment shader. No doubt there's a way I can optimize this, but I haven't tried yet. The zoom blur uses a 9-hit Gaussian blur where the direction and per-sample offset distance vary as a function of the placement of the pixel vs. the center of the blur.
It uses the following fragment shader (and a standard passthrough vertex shader):
varying highp vec2 textureCoordinate;
uniform sampler2D inputImageTexture;
uniform highp vec2 blurCenter;
uniform highp float blurSize;
void main()
{
// TODO: Do a more intelligent scaling based on resolution here
highp vec2 samplingOffset = 1.0/100.0 * (blurCenter - textureCoordinate) * blurSize;
lowp vec4 fragmentColor = texture2D(inputImageTexture, textureCoordinate) * 0.18;
fragmentColor += texture2D(inputImageTexture, textureCoordinate + samplingOffset) * 0.15;
fragmentColor += texture2D(inputImageTexture, textureCoordinate + (2.0 * samplingOffset)) * 0.12;
fragmentColor += texture2D(inputImageTexture, textureCoordinate + (3.0 * samplingOffset)) * 0.09;
fragmentColor += texture2D(inputImageTexture, textureCoordinate + (4.0 * samplingOffset)) * 0.05;
fragmentColor += texture2D(inputImageTexture, textureCoordinate - samplingOffset) * 0.15;
fragmentColor += texture2D(inputImageTexture, textureCoordinate - (2.0 * samplingOffset)) * 0.12;
fragmentColor += texture2D(inputImageTexture, textureCoordinate - (3.0 * samplingOffset)) * 0.09;
fragmentColor += texture2D(inputImageTexture, textureCoordinate - (4.0 * samplingOffset)) * 0.05;
gl_FragColor = fragmentColor;
}
Note that both of these blurs are hardcoded at 9 samples for performance reasons. This means that at larger blur sizes, you'll start to see artifacts from the limited samples here. For larger blurs, you'll need to run these filters multiple times or extend them to support more Gaussian samples. However, more samples will lead to much slower rendering times because of the limited texture sampling bandwidth on iOS devices.
CoreImage has a Motion Blur filter.
It's called CIMotionBlur... http://developer.apple.com/library/mac/#documentation/GraphicsImaging/Reference/CoreImageFilterReference/Reference/reference.html#//apple_ref/doc/filter/ci/CIMotionBlur