I'm trying to write a shader to colorize a custom geometry in SceneKit. I want to set the colors so that horizontal surfaces that are facing up are white, horizontal surfaces that are facing downward are black, and all other surfaces in between are shades of gray based on their orientation. I used the material's surface entry point to call the following shader
vec4 normal = vec4(_surface.normal, 1.0);
// Assume value is [-1, 1], scale to [0, 1]
float color = normal.z * 0.5 + 0.5;
_surface.diffuse = vec4(color, color, color, 1.0);
It appears that the normal.z is relative to the camera (or view). I assume I need to transform the value, so that it's in another space. I tried multiple transforms (and combinations of transforms) such as u_inverseViewTransform, but the results all seem to be in view space. Does anyone know how to colorize a geometry based on the orientation of its surfaces?
Since _surface.normal is a vector in view space, you have to transform the vector to world space.
The transformation of a point form world space to view space is done by u_viewTransform, so the inverse operation (from view space to world space) can be done by u_inverseViewTransform.
_surface.normal is a direction vector, but not a point. A vector has to be transformed by the normal matrix, which is the transposed, inverse of the upper left 3*3 matrix of a 4*4 matrix:
transpose(inverse(mat3(u_viewTransform)))
See Why is the transposed inverse of the model view matrix used to transform the normal vectors?, Why transforming normals with the transpose of the inverse of the modelview matrix?
But since u_viewTransform and u_inverseViewTransform are Orthogonal matrices the transposed, inverse can be skipped, because the inverse matrix and the transposed matrix are equal. See In which cases is the inverse matrix equal to the transpose?.
This follows you have to transform by mat3(u_inverseViewTransform):
vec3 normal = mat3(u_inverseViewTransform) * _surface.normal;
float color = normal.z * 0.5 + 0.5;
_surface.diffuse = vec4(color, color, color, 1.0);
Related
I am currently studying shadow mapping, and my biggest issue right now is the transformations between spaces. This is my current working theory/steps.
Pass 1:
Get depth of pixel from camera, store in depth buffer
Get depth of pixel from light, store in another buffer
Pass 2:
Use texture coordinate to sample camera's depth buffer at current pixel
Convert that depth to a view space position by multiplying the projection coordinate with invProj matrix. (also do a perspective divide).
Take that view position and multiply by invV (camera's inverse view) to get a world space position
Multiply world space position by light's viewProjection matrix.
Perspective divide that projection-space coordinate, and manipulate into [0..1] to sample from light depth buffer.
Get current depth from light and closest (sampled) depth, if current depth > closest depth, it's in shadow.
Shader Code
Pass1:
PS_INPUT vs(VS_INPUT input) {
output.pos = mul(input.vPos, mvp);
output.cameraDepth = output.pos.zw;
..
float4 vPosInLight = mul(input.vPos, m);
vPosInLight = mul(vPosInLight, light.viewProj);
output.lightDepth = vPosInLight.zw;
}
PS_OUTPUT ps(PS_INPUT input){
float cameraDepth = input.cameraDepth.x / input.cameraDepth.y;
//Bundle cameraDepth in alpha channel of a normal map.
output.normal = float4(input.normal, cameraDepth);
//4 Lights in total -- although only 1 is active right now. Going to use r/g/b/a for each light depth.
output.lightDepths.r = input.lightDepth.x / input.lightDepth.y;
}
Pass 2 (Screen Quad):
float4 ps(PS_INPUT input) : SV_TARGET{
float4 pixelPosView = depthToViewSpace(input.texCoord);
..
float4 pixelPosWorld = mul(pixelPosView, invV);
float4 pixelPosLight = mul(pixelPosWorld, light.viewProj);
float shadow = shadowCalc(pixelPosLight);
//For testing / visualisation
return float4(shadow,shadow,shadow,1);
}
float4 depthToViewSpace(float2 xy) {
//Get pixel depth from camera by sampling current texcoord.
//Extract the alpha channel as this holds the depth value.
//Then, transform from [0..1] to [-1..1]
float z = (_normal.Sample(_sampler, xy).a) * 2 - 1;
float x = xy.x * 2 - 1;
float y = (1 - xy.y) * 2 - 1;
float4 vProjPos = float4(x, y, z, 1.0f);
float4 vPositionVS = mul(vProjPos, invP);
vPositionVS = float4(vPositionVS.xyz / vPositionVS.w,1);
return vPositionVS;
}
float shadowCalc(float4 pixelPosL) {
//Transform pixelPosLight from [-1..1] to [0..1]
float3 projCoords = (pixelPosL.xyz / pixelPosL.w) * 0.5 + 0.5;
float closestDepth = _lightDepths.Sample(_sampler, projCoords.xy).r;
float currentDepth = projCoords.z;
return currentDepth > closestDepth; //Supposed to have bias, but for now I just want shadows working haha
}
CPP Matrices
// (Position, LookAtPos, UpDir)
auto lightView = XMMatrixLookAtLH(XMLoadFloat4(&pos4), XMVectorSet(0,0,0,1), XMVectorSet(0,1,0,0));
// (FOV, AspectRatio (1000/680), NEAR, FAR)
auto lightProj = XMMatrixPerspectiveFovLH(1.57f , 1.47f, 0.01f, 10.0f);
XMStoreFloat4x4(&_cLightBuffer.light.viewProj, XMMatrixTranspose(XMMatrixMultiply(lightView, lightProj)));
Current Outputs
White signifies that a shadow should be projected there. Black indicates no shadow.
CameraPos (0, 2.5, -2)
CameraLookAt (0, 0, 0)
CameraFOV (1.57)
CameraNear (0.01)
CameraFar (10.0)
LightPos (0, 2.5, -2)
LightLookAt (0, 0, 0)
LightFOV (1.57)
LightNear (0.01)
LightFar (10.0)
If I change the CameraPosition to be (0, 2.5, 2), basically just flipped on the Z axis, this is the result.
Obviously a shadow shouldn't change its projection depending on where the observer is, so I think I'm making a mistake with the invV. But I really don't know for sure. I've debugged the light's projView matrix, and the values seem correct - going from CPU to GPU. It's also entirely possible I've misunderstood some theory along the way because this is quite a tricky technique for me.
Aha! Found my problem. It was a silly mistake, I was calculating the depth of pixels from each light, but storing them in a texture that was based on the view of the camera. The following image should explain my mistake better than I can with words.
For future reference, the solution I decided was to scrap my idea for storing light depths in texture channels. Instead, I basically make a new pass for each light, and bind a unique depth-stencil texture to render the geometry to. When I want to do light calculations, I bind each of the depth textures to a shader resource slot and go from there. Obviously this doesn't scale well with many lights, but for my student project where I'm only required to have 2 shadow casters, it suffices.
_context->DrawIndexed(indexCount, 0, 0); //Draw to regular render target
_sunlight->use(1, _context); //Use sunlight shader (basically just runs a Vertex Shader & Null Pixel shader so depth can be written to depth map)
_sunlight->bindDSVSetNullRenderTarget(_context);
_context->DrawIndexed(indexCount, 0, 0); //Draw to sunlight depth target
bindDSVSetNullRenderTarget(ctx){
ID3D11RenderTargetView* nullrv = { nullptr };
ctx->OMSetRenderTargets(1, &nullrv, _sunlightDepthStencilView);
}
//The purpose of setting a null render target before doing the draw call is
//that a draw call with only a depth target bound is much faster.
//(At least I believe so, from my reading online)
In OpenCV how do you calculate the average gradient strength in a Mat and the average gradient direction?
I have sourced the below methods by googling but I want to confirm I am actually doing this correctly before moving onto the next step.
Is this correct?
Mat img = imread('foo.png', CV_8UC); // read image as grayscale single channel
// Calculate the mean intensity and the std deviation
// Any errors here or am I doing this correctly?
Scalar sMean, sStdDev;
meanStdDev(src, sMean, sStdDev);
double mean = sMean[0];
double stddev = sStdDev[0];
// Calculate the average gradient magnitude/strength across the image
// Any errors here or am I doing this correctly?
Mat dX, dY, magnitude;
Sobel(src, dX, CV_32F, 1, 0, 1);
Sobel(src, dY, CV_32F, 0, 1, 1);
magnitude(dX, dY, magnitude);
Scalar sMMean, sMStdDev;
meanStdDev(magnitude, sMMean, sMStdDev);
double magnitudeMean = sMMean[0];
double magnitudeStdDev = sMStdDev[0];
// Calculate the average gradient direction across the image
// Any errors here or am I doing this correctly?
Scalar avgHorizDir = mean(dX);
Scalar avgVertDir = mean(dY);
double avgDir = atan2(-avgVertDir[0], avgHorizDir[0]);
float blurriness = cv::videostab::calcBlurriness(src); // low values = sharper. High values = blurry
Technically those are the correct ways of obtaining the two averages.
The way you compute mean direction uses weighted directional statistics, meaning that pixels without a strong gradient have less influence on the average.
However, for most images this average direction is not very meaningful, as there exist edges in all directions and cancel out.
If your image is of a single edge, then this will work great.
If your image has lines in it, containing edges in opposite directions, this will not work. In this case, you want to average the double angle (average orientations). The obvious way of doing this is to compute the direction per pixel as an angle, double them, then use directional statistics to average (ie convert back to vectors and average those). Doubling the angle causes opposite directions to be mapped to the same value, thus averaging doesn’t cancel these out.
Another simple way to average orientations is to take the average of the tensor field obtained by the outer product of the gradient field with itself, and determine the direction of the eigenvector corresponding to the largest eigenvalue. The tensor field is obtained as follows:
Mat Sxx = dX * dX;
Mat Syy = dY * dY;
Mat Sxy = dX * dY;
This should then be averaged:
Scalar mSxx = mean(sXX);
Scalar mSyy = mean(sYY);
Scalar mSxy = mean(sXY);
These values form a 2x2 real-valued symmetric matrix:
| mSxx mSxy |
| mSxy mSyy |
It is relatively straight-forward to determine its eigendecomposition, and can be done analytically. I don’t have the equations on hand right now, so I’ll leave it as an exercise to the reader. :)
im having problem to get specular lighting to work. It looks like im having some kind of bug in my application which im not able to trace.
Light is coming from the front (screenshots shows camera looking at -z direction, on the left is -x).
A simple specular output shows the following failure:
Code used:
float3 n = normalize(input.normal); // world space normal, mul(float4(normal, 0.0), modelMatrix)
float3 l = normalize(-sDirection); // constant direction (like 0.7, -0.8, -0.7)
float3 v = normalize(viewPos.xyz - input.vertexWorldSpace.xyz); // viewpos = world space camera position
float3 LightReflect = normalize(reflect(n,l));
float SpecularFactor = dot(v, LightReflect);
color = float4(SpecularFactor, SpecularFactor, SpecularFactor, 1.0);
To check for possible variable errors i checked the input.VertexWorldSpace:
to also check lightdirection and normal, i checked the diffuse term:
and the camera to vertex view vector (v):
For me, all parts look fine, but still the specular gets black at origin(0,0,0) and perpendicular to the light direction.
Ive also made a gif showing what happens with the view direction vector at origin(0, 0, 0)
http://imgur.com/2YlqcGP
And another gif showing camera position and where the specular goes black:
http://i.imgur.com/ajUaekA.gifv
Am i using a wrong calculation for v?
Ok, the problem was with my buffer alignment: https://msdn.microsoft.com/en-us/library/windows/desktop/bb509632(v=vs.85).aspx
Instead of passing
Matrix
Matrix
Vec3
Vec3
hlsl wants 16 byte alignment vectors
so i had to use
Matrix
Vec3
float1 // 16 byte alignment
Vec3
float1 // 16 byte alignment
I'm studied the pARK example project (http://developer.apple.com/library/IOS/#samplecode/pARk/Introduction/Intro.html#//apple_ref/doc/uid/DTS40011083) so I can apply some of its fundamentals in an app i'm working on. I understand nearly everything, except:
The way it has to calculate if a point of interest must appear or not. It gets the attitude, multiply it with the projection matrix (to get the rotation in GL coords?), then multiply that matrix with the coordinates of the point of interest and, at last, look at the last coordinate of that vector to find out if the point of interest must be shown. Which are the mathematic fundamentals of this?
Thanks a lot!!
I assume you are referring to the following method:
- (void)drawRect:(CGRect)rect
{
if (placesOfInterestCoordinates == nil) {
return;
}
mat4f_t projectionCameraTransform;
multiplyMatrixAndMatrix(projectionCameraTransform, projectionTransform, cameraTransform);
int i = 0;
for (PlaceOfInterest *poi in [placesOfInterest objectEnumerator]) {
vec4f_t v;
multiplyMatrixAndVector(v, projectionCameraTransform, placesOfInterestCoordinates[i]);
float x = (v[0] / v[3] + 1.0f) * 0.5f;
float y = (v[1] / v[3] + 1.0f) * 0.5f;
if (v[2] < 0.0f) {
poi.view.center = CGPointMake(x*self.bounds.size.width, self.bounds.size.height-y*self.bounds.size.height);
poi.view.hidden = NO;
} else {
poi.view.hidden = YES;
}
i++;
}
}
This is performing an OpenGL like vertex transformation on the places of interest to check if they are in a viewable frustum. The frustum is created in the following line:
createProjectionMatrix(projectionTransform, 60.0f*DEGREES_TO_RADIANS, self.bounds.size.width*1.0f / self.bounds.size.height, 0.25f, 1000.0f);
This sets up a frustum with a 60 degree field of view, a near clipping plane of 0.25 and a far clipping plane of 1000. Any point of interest that is further away than 1000 units will then not be visible.
So, to step through the code, first the projection matrix that sets up the frustum, and the camera view matrix, which simply rotates the object so it is the right way up relative to the camera, are multiplied together. Then, for each place of interest, its location is multiplied by the viewProjection matrix. This will project the location of the place of interest into the view frustum, applying rotation and perspective.
The next two lines then convert the transformed location of the place into whats known as normalized device coordinates. The 4 component vector needs to be collapsed to 3 dimensional space, this is achieved by projecting it onto the plane w == 1, by dividing the vector by its w component, v[3]. It is then possible to determine if the point lies within the projection frustum by checking if its coordinates lie in the cube with side length 2 with origin [0, 0, 0]. In this case, the x and y coordinates are being biased from the range [-1 1] to [0 1] to match up with the UIKit coordinate system, by adding 1 and dividing by 2.
Next, the v[2] component, z, is checked to see if it is greater than 0. This is actually incorrect as it has not been biased, it should be checked to see if it is greater than -1. This will detect if the place of interest is in the first half of the projection frustum, if it is then the object is deemed visible and displayed.
If you are unfamiliar with vertex projection and coordinate systems, this is a huge topic with a fairly steep learning curve. There is however a lot of material online covering it, here are a couple of links to get you started:
http://www.falloutsoftware.com/tutorials/gl/gl0.htm
http://www.opengl.org/wiki/Vertex_Transformation
Good luck//
I am working with WebGL using three.js and I have an image that I want to project onto the (inner) surface of a sphere. The problem I am facing is how to limit the extent of that mapping to a horizontal and vertical field of view. Imagine projecting an image from the centre of a sphere onto a rectangular section of it.
I suspect I can do this one of 2 ways, but am unsure about how to do either...
1) Map the texture onto the sphere based on the field of view angles. Mapping the image straight onto the sphere as below does a 360x180 degree wrap. Is there a UV mapping trick involved, or some other technique available?
var sphere = new THREE.Mesh(
new THREE.SphereGeometry(radius, 60, 40),
new THREE.MeshBasicMaterial(
{ map: THREE.ImageUtils.loadTexture( filename ) }
)
);
2) Chop up a sphere so that it only has the subset of the surface covered by the angles given (ie intersecting with a rectangular pyramid), or producing an equivalent curved surface. Any ideas?
The easiest way to scale down the projection is to tweak the UV coords in the fragment shader:
// how large the projection should be
uniform vec2 uScale;
...
// this is the color for pixels outside the mapped texture
vec4 texColor = vec4(0.0, 0.0, 0.0, 1.0);
vec2 scale = vec2(1.0/uScale.s, 1.0/uScale.t);
vec2 mappedUv = vUv*scale + vec2(0.5,0.5)*(vec2(1.0,1.0)-scale);
// if the mapped uv is inside the texture area, read from texture
if (mappedUv.s >= 0.0 && mappedUv.s <= 1.0 &&
mappedUv.t >= 0.0 && mappedUv.t <= 1.0) {
texColor = texture2D(map, mappedUv);
}
For THREE.SphereGeometry UVs the full field of view in radians is 2pi for x and pi for y. The scale factor for a reduced field is vec2(fovX/2pi, fovY/pi).
You can also do the UV scaling in the vertex shader. Other ways are to copypaste https://github.com/mrdoob/three.js/blob/master/src/extras/geometries/SphereGeometry.js and change the generated UVs to match your scaling factors (uv*1/scale + 0.5*(1-1/scale))
Lemme know if this helps.