I'm trying to draw a 3D line using metalkit in iOS, I added two vertices
[{ .position = { 1, 1, -1, 1 }, .color = { 0, 0, 1, 1 } },
{ .position = { 1, 1, 1, 1 }, .color = { 0, 0, 1, 1 } }]
I'm applying modal matrix with rotation angle.
const matrix_float4x4 modelMatrix = zRot;
But the line is rotating continuously, i tried using translation matrix but no difference i observe.
Actually my task is to draw a line in 3D for augmented reality, placing markers using latitude and longitude with path along them.
As a first step i'm trying to draw a 3D line using metal. Plz Refer any example for drawing 3D line using metalkit most of the examples are in triangles.
Related
Sample code: https://developer.apple.com/documentation/arkit/visualizing_a_point_cloud_using_scene_depth
In the code, when unprojecting depthmap into world point, we are using a positive z value(depth value). But in my understanding, ARKit uses right-handed coordinate system which means points with positive z value are behind the camera. So maybe we need to do some extra work to align the coordinate system(using rotateToARCamera matrix?). But I cannot understand why we need to flip both Y and Z plane.
static func makeRotateToARCameraMatrix(orientation: UIInterfaceOrientation) -> matrix_float4x4 {
// flip to ARKit Camera's coordinate
let flipYZ = matrix_float4x4(
[1, 0, 0, 0],
[0, -1, 0, 0],
[0, 0, -1, 0],
[0, 0, 0, 1] )
let rotationAngle = Float(cameraToDisplayRotation(orientation: orientation)) * .degreesToRadian
return flipYZ * matrix_float4x4(simd_quaternion(rotationAngle, Float3(0, 0, 1)))
}
Update: I guess the key point is the coordinate system used for camera intrinsics matrix's pin-hole model has an inverse direction compared to the normal camera space in ARKit.
Depth Map is a coordinate system where the Y coordinate is smaller at the top and larger at the bottom like image data, but ARKit is a coordinate system where the Y coordinate is smaller from the bottom and larger at the top.
For this reason, I think it is necessary to invert the Y coordinate.
Apples Metal hello 2D triangle example doesn't appear to be 2D in the way I had hoped.
I was expecting to have 1 on x and 1 on y to be exactly 1 pixel and for it to start at the top left.
I would have thought that 2D implies flat and with no concept of depth it naturally follows to have 1 unit to map to 1 pixel, how can I fix the example to work in the way I expected it to? General concepts of coarse, no need to actually produce the code unless you really like unicorns in which case please do; so I can communicate to the world my brilliance.
https://developer.apple.com/documentation/metal/hello_triangle
static const AAPLVertex triangleVertices[] =
{
// 2D positions, RGBA colors
{ { 0, 0 }, { 1, 0, 0, 1 } },
{ { 0, 4 }, { 0, 1, 0, 1 } },
{ { 4, 0 }, { 0, 0, 1, 1 } },
{ { 4, 4 }, { 0, 1, 0, 1 } },
};
These coordinates as a line strip produces a 4 by 5 px N!
Update
I was unable to resolve/understand why a 4 by 5 px line strip is drawn. I believe the line strip algorithm is incorrect.
Triangle strip with expected result
vs
Same but line strip with unexpected results
Consider this line strip 3 vertex corner:
static const AAPLVertex triangleVertices[] =
{
// 2D positions, RGBA colors
{ { 0, 4 }, { 1, 0, 0, 1 } },
{ { 0, 0 }, { 0, 1, 0, 1 } },
{ { 4, 0 }, { 0, 0, 1, 1 } },
};
corner as line strip image
It seems you basically want to specify your initial vertex positions in window coordinates. That will get you the 1:1 pixel-to-unit mapping that you're after.
The job here is to come up with a sequence of transformations that allows you to specify vertices in window space, while honoring the fact that vertex positions returned from your vertex function must be in clip space. The inverse of this transformation is applied during rasterization, so we're trying to cancel it out.
We'll ignore depth. Clip space ranges from -1 to 1 in both X and Y directions. So we want to map the origin of our space (0, 0) to (-1, 1) in clip space. Similarly, we want to map the bottom-right (width, height) of our space to (1, -1) in clip space.
We can get most of the way there by multiplying the pixel coordinates by 2, subtracting the viewport size, and finally dividing by the viewport size. This replaces the computation of the clip space coordinates in the example's vertex function:
out.clipSpacePosition.xy = ((pixelSpacePosition * 2) - viewportSize) / viewportSize;
In this space, Y increases downward, but the Y axis in clip space increases upward, so we need to flip it:
out.clipSpacePosition.y *= -1;
(editing the question completely as i am trying a different approach)
I am trying to find the corners of a plain paper sheet which can be partially obstructed.
I am using houghlines now to get the lines and then calculating their intersections, this approach is robust and gives good results even if the paper is partially obstructed.
Problems:
1) There are multiple intersection points very closeby, how do i merge them fast as i am displaying the results in realtime.
2) As there might be some points detected outside the paper (due to some noise of some other object) how do i get a polygon from the intersection points.
GaussianBlur(grayscaleMat, grayscaleMat, cvSize(11,11), 0);//change from median blur to gaussian for more accuracy of square detection
cv::Canny(grayscaleMat, grayscaleMat, 50, 150, 3);
cv::Mat color;
cv::cvtColor(grayscaleMat, color, cv::COLOR_GRAY2BGR);
vector<cv::Vec2f> lines;
HoughLines(grayscaleMat, lines, 1, CV_PI/180, 50, 0, 0 );
if(lines.size() == 0) {
return image;
}
////////////////////////////////////
// Start geometery
//merge close lines
std::vector<cv::Vec2f> linesMerged = [ATOpenCVGeometery mergeCloseLines:lines];
//find intersections
std::vector<cv::Point2f> points = [ATOpenCVGeometery intersectionsBetweenLines:linesMerged];
for(int i = 0;i < linesMerged.size();i++) {
[self drawLine:image vec2f:linesMerged[i] color:cv::Scalar(0,0,255)];
}
for(int i = 0;i < points.size();i++) {
cv::circle(image, points[i], 4, cv::Scalar(255,0,0));
}
I have vertices of some surfaces that I draw on the canvas using drawArrays(gl.TRIANGLES,...). I need to draw these surfaces for a particular camera viewpoint and hence all 3d points are projected into 2d and I download the final image using toDataUrl. Here is the downloaded image:
I used gl.readPixels later to retrieve the data for everypixel.
For all the edge vertices, I have the information for the normals. Just like how I got the color for every pixel in the 2d images, I want to get the normals at every pixel for 2d image. Since I only have the normals at the edge vertices, I decided to render the normals the same way I rendered the above image and decided to use gl.readpixels. This is not working. Here is the relevant code:
This is the function from which the drawOverlayTriangeNormals is called. The drawOverlayTriangles function (not visible in this post) was used to produce the image shown above.
//Saving BIM
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
gl.vertexAttrib1f(shaderProgram.aIsDepth, 0.0);
drawOverlayTriangles();
saveBlob('element');
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
gl.vertexAttrib1f(shaderProgram.aIsDepth, 0.0);
drawOverlayTrianglesNormals();
saveBlob('element');
var pixels = new Uint8Array(glCanvas.width*glCanvas.height*4);
gl.readPixels(0, 0, glCanvas.width, glCanvas.height, gl.RGBA, gl.UNSIGNED_BYTE,pixels);
pixels = new Float32Array(pixels.buffer);
}
This is the drawOverlayTrianglesNormals function:
function drawOverlayTrianglesNormals()
{
if (overlay.numElements <= 0)
return;
//Creating the matrix for normal transform
var normal_matrix = mat4.create();
var u_Normal_Matrix = mat4.create();
mat4.invert(normal_matrix,pMVMatrix);
mat4.transpose(u_Normal_Matrix,normal_matrix);
gl.enable(gl.DEPTH_TEST);
gl.enableVertexAttribArray(shaderProgram.aVertexPosition);
gl.enableVertexAttribArray(shaderProgram.aVertexColor);
gl.enableVertexAttribArray(shaderProgram.aNormal);
gl.vertexAttrib1f(shaderProgram.aIsNormal, 1.0);
//Matrix upload
gl.uniformMatrix4fv(shaderProgram.uMVMatrix, false, pMVMatrix);
gl.uniformMatrix4fv(shaderProgram.uPMatrix, false, perspM);
gl.uniformMatrix4fv(shaderProgram.uNMatrix, false, u_Normal_Matrix);
//Create normals buffer
normals_buffer = gl.createBuffer();
for (var i = 0; i < overlay.numElements; i++) {
// Upload overlay vertices
gl.bindBuffer(gl.ARRAY_BUFFER, overlayVertices[i]);
gl.vertexAttribPointer(shaderProgram.aVertexPosition, 3, gl.FLOAT, false, 0, 0);
// Upload overlay colors
gl.bindBuffer(gl.ARRAY_BUFFER, overlayTriangleColors[i]);
gl.vertexAttribPointer(shaderProgram.aVertexColor, 4, gl.FLOAT, false, 0, 0);
var normal_vertex = [];
//Upload Normals
var normals_element = overlay.elementNormals[i];
for( var j=0; j< overlay.elementNumVertices[i]; j++)
{
var x = normals_element[3*j+0];
var y = normals_element[3*j+1];
var z = normals_element[3*j+2];
var length = Math.sqrt(x*x + y*y + z*z);
normal_vertex[3*j+0] = x/length;
normal_vertex[3*j+1] = y/length;
normal_vertex[3*j+2] = z/length;
}
gl.bindBuffer(gl.ARRAY_BUFFER, normals_buffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(normal_vertex),gl.STATIC_DRAW);
gl.vertexAttribPointer(shaderProgram.aNormal, 3, gl.FLOAT, false, 0, 0);
// Draw overlay
gl.drawArrays(gl.TRIANGLES, 0, overlay.elementNumVertices[i]);
}
gl.disableVertexAttribArray(shaderProgram.aVertexPosition);
gl.disableVertexAttribArray(shaderProgram.aVertexColor);
gl.vertexAttrib1f(shaderProgram.aisdepth, 0.0);
}
Below is the relevant vertex shader code:
void main(void) {
gl_PointSize = aPointSize;
gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0);
position_1 = gl_Position;
vColor = aVertexColor;
vIsDepth = aIsDepth;
vIsNormal = aIsNormal;
vHasTexture = aHasTexture;
normals = uNMatrix*vec4(aNormal,1.0);
if (aHasTexture > 0.5)
vTextureCoord = aTextureCoord;
}
Fragment shader:
if (vIsNormal > 0.5)
{
gl_FragColor = vec4(normals.xyz*0.5+0.5,1);
}
}
Right now my output is the same image in grayscale. I am not sure what is going wrong. I felt this method makes sense but seems a little round about.
I'm not entirely sure I understand what you're trying to do, but it seems like you just want to be able to access the normals for calculating lighting effects, so let me try to answer that.
DO NOT use gl.readPixels()! This is primarily for click interactions and stuff, or modifying small numbers of pixels. Using this must be extremely inefficient, since you have to draw the pixels, then read them, then redraw them after calculating their appropriate lighting. The wonderful thing about WebGL is that it allows you to do all this from the beginning: the fragment shader will interpolate the information it's given to smoothly draw effects between two adjacent vertices.
Most of lighting depends comparing the surface normal to the direction of the light (as you seem to understand, judging by one of your comments). See Phong shading.
Now, you mentioned that you want the normals of all the rendered points, not just at the vertices. BUT the vertices' normals will be identical to the normals at every point on the surface, so you don't even need anything more than the vertices' normals. This is because all WebGL knows how to draw is triangles (I believe), which are flat, or planar, surfaces. And since every point on a plane has the same normal as any other point, you only really need one normal to know all of the normals!
Since it looks like all you're trying to draw are cylinders and rectangular prisms, it ought to be simple to specify the normals the objects you create. The normals for the rectangular prisms are trivial, but so are those of the cylinder: the normals are parallel to the line going from the axis of the cylinder to the surface.
And since WebGL's fragment shader interpolates any varying variables you pass it between adjacent vertices, you can tell it to interpolate these normals smoothly across vertices, to achieve the smooth lighting seen in the Phong shadin page! :D
I am trying to pre multiply a Homography matrix before I send it it the warpperspective function, but I cannot figure out how to do this. I am trying to use gemm for multiplying the matrices. Also How do you specify an element (like HomOffset(0,0)) in a matrix obj then multiply it by a scalar? I have been reading the opencv documentation but did not come across this. Code is below. Thanks in advance.
cv:: Mat Hom = cv::findHomography(scene,obj, CV_RANSAC);
cv:: Mat HomOffset[3][3] = {
{ 1, 0, 25 },
{ 0, 1, 25 },
{ 0, 0, 1 }
};
error for declartion of HomOffSet code is int to cv:: Mat is ambigious
gemm(Hom,HomOffset,1,0,0,H);
Multiple errors for the gemm function.
you need to assign your Matrix's values (HomOffset) correctly. Use at operator: see it here