Screen space raycasting - ios

I am using OpenGL ES 2.0 and the latest iOS.
I have a 3D model that I wish the user to be able to select different parts of the model by tapping on the screen. I have found this tutorial on converting a pixel-space screen coordinate to a world-space ray, and have implemented a ray-AABB intersection test to determine the intersection portion of the model.
I get some hits on the model at seemingly random sections of the model. So I need to debug this feature, but I don't really know where to start.
I can't exactly draw a line representing the ray (since it is coming out of the camera it will appear as a point), so I can see a couple of ways of debugging this:
Check the bounding boxes of the model sections. So is there an easy way with OGL ES to draw a bounding box given a min and max point?
draw some 3D object along the path of the ray. This seems more complicated.
Actually debug the raycast and intersection code. This seems like the hardest to accomplish since the algorithms are fairly well known (I took the intersection test straight ouf of my Real-Time Collision Detection book).
If anyone can help, or wants me to post some code, I could really use it.
Here is my code for converting to world space:
- (IBAction)tappedBody:(UITapGestureRecognizer *)sender
{
if ( !editMode )
{
return;
}
CGPoint tapPoint = [sender locationOfTouch:0 inView:self.view];
const float tanFOV = tanf(GLKMathDegreesToRadians(65.0f*0.5f));
const float width = self.view.frame.size.width,
height = self.view.frame.size.height,
aspect = width/height,
w_2 = width * 0.5,
h_2 = height * 0.5;
CGPoint screenPoint;
screenPoint.x = tanFOV * ( tapPoint.x / w_2 - 1 ) / aspect;
screenPoint.y = tanFOV * ( 1.0 - tapPoint.y / h_2 );
GLKVector3 nearPoint = GLKVector3Make(screenPoint.x * NEAR_PLANE, screenPoint.y * NEAR_PLANE, NEAR_PLANE );
GLKVector3 farPoint = GLKVector3Make(screenPoint.x * FAR_PLANE, screenPoint.y * FAR_PLANE, FAR_PLANE );
GLKVector3 nearWorldPoint = GLKMatrix4MultiplyVector3( _invViewMatrix, nearPoint );
GLKVector3 farWorldPoint = GLKMatrix4MultiplyVector3( _invViewMatrix, farPoint );
GLKVector3 worldRay = GLKVector3Subtract(farWorldPoint, nearWorldPoint);
NSLog(#"Model matrix: %#", NSStringFromGLKMatrix4(_modelMatrix));
worldRay = GLKVector4Normalize(worldRay);
[male intersectWithRay:worldRay fromStartPoint:nearWorldPoint];
for ( int i =0; i < 3; ++i )
{
touchPoint[i] = nearWorldPoint.v[i];
}
}
And here's how I get the matrices:
- (void)update
{
// _rotation = 0;
float aspect = fabsf(self.view.bounds.size.width / self.view.bounds.size.height);
GLKMatrix4 projectionMatrix = GLKMatrix4MakePerspective(GLKMathDegreesToRadians(65.0f), aspect, NEAR_PLANE, FAR_PLANE);
self.effect.transform.projectionMatrix = projectionMatrix;
GLKMatrix4 baseModelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -5.0f);
// Compute the model view matrix for the object rendered with ES2
_viewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, 0.0f);
_modelMatrix = GLKMatrix4Rotate(baseModelViewMatrix, _rotation, 0.0f, 1.0f, 0.0f);
_modelViewMatrix = GLKMatrix4Rotate(_viewMatrix, _rotation, 0.0f, 1.0f, 0.0f);
_modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, _modelViewMatrix);
_invViewMatrix = GLKMatrix4Invert(_viewMatrix, NULL);
_invMVMatrix = GLKMatrix4Invert(_modelViewMatrix, NULL);
_normalMatrix = GLKMatrix3InvertAndTranspose(GLKMatrix4GetMatrix3(_modelViewMatrix), NULL);
_modelViewProjectionMatrix = GLKMatrix4Multiply(projectionMatrix, _modelViewMatrix);
male.modelTransform = _modelMatrix;
if ( !editMode )
{
_rotation += self.timeSinceLastUpdate * 0.5f;
}
}

I can't exactly draw a line representing the ray (since it is coming
out of the camera it will appear as a point)
DonĀ“t discard this so soon. Isn't it that what you are trying to test? I mean, That would be so if you are unprojecting things right. But if it you have a bug, it won't.
I'll go with this first... if you see a point just under your finger the conversion is right, and you can start investigating the other options you pointed out, which are more complex.

I'm not sure why this fixed the problem I was having but changing
screenPoint.x = tanFOV * ( tapPoint.x / w_2 - 1 ) / aspect;
to
screenPoint.x = tanFOV * ( tapPoint.x / w_2 - 1 ) * aspect;
and
GLKVector4 nearPoint = GLKVector4Make(screenPoint.x * NEAR_PLANE, screenPoint.y * NEAR_PLANE, NEAR_PLANE, 1 );
GLKVector4 farPoint = GLKVector4Make(screenPoint.x * FAR_PLANE, screenPoint.y * FAR_PLANE, FAR_PLANE, 1 );
to
GLKVector4 nearPoint = GLKVector4Make(screenPoint.x * NEAR_PLANE, screenPoint.y * NEAR_PLANE, -NEAR_PLANE, 1 );
GLKVector4 farPoint = GLKVector4Make(screenPoint.x * FAR_PLANE, screenPoint.y * FAR_PLANE, -FAR_PLANE, 1 );
seems to have fixed the raycasting issue. Still not sure why my view matrix seems to be indicating that the camera is looking down the positive Z-axis, but my objects are translated negative along the z axis

Related

Line Pixellating

Using Metal I am drawing line using Bezier Curves using four points. I am using nearly 1500 triangles for the lines. The line is Pixellated. How can i reduce pixellated.
vertex VertexOutBezier bezier_vertex(constant BezierParameters *allParams[[buffer(0)]],
constant GlobalParameters& globalParams[[buffer(1)]],
uint vertexId [[vertex_id]],
uint instanceId [[instance_id]])
{
float t = (float) vertexId / globalParams.elementsPerInstance;
rint(t);
BezierParameters params = allParams[instanceId];
float lineWidth = (1 - (((float) (vertexId % 2)) * 2.0)) * params.lineThickness;
float2 a = params.a;
float2 b = params.b;
float cx = distance(a , b);
float2 p1 = params.p1 * 3.0; // float2 p1 = params.p1 * 3.0;
float2 p2 = params.p2 * 3.0; // float2 p2 = params.p2 * 3.0;
float nt = 1.0f - t;
float nt_2 = nt * nt;
float nt_3 = nt_2 * nt;
float t_2 = t * t;
float t_3 = t_2 * t;
// Calculate a single point in this Bezier curve:
float2 point = a * nt_3 + p1 * nt_2 * t + p2 * nt * t_2 + b * t_3;
float2 tangent = -3.0 * a * nt_2 + p1 * (1.0 - 4.0 * t + 3.0 * t_2) + p2 * (2.0 * t - 3.0 * t_2) + 3 * b * t_2;
tangent = (float2(-tangent.y , tangent.x ));
VertexOutBezier vo;
vo.pos.xy = point + (tangent * (lineWidth / 2.0f));
vo.pos.zw = float2(0, 1);
vo.color = params.color;
return vo;
}
You need to enable MSAA (multisample anti-aliasing). How you do this depends on your exact Metal view configuration, but the easiest way is if you're using MTKView. To enable MSAA in an MTKView, all you have to do is:
metalView.sampleCount = 4
Then, when you configure your MTLRenderPipelineDescriptor before calling makeRenderPipelineState(), add the following:
pipelineDescriptor.sampleCount = 4
This should greatly improve the quality of your curves and reduce pixelation. It does come with a performance cost however, as the GPU has to do substantially more work to render your frame.

HLSL Geometry Shader Thick Lines DirectX

I try to draw a thick lined Sinus curve using this approach : https://forum.libcinder.org/topic/smooth-thick-lines-using-geometry-shader
I tried to port it to HLSL geometry shader:
Setting the dimension to fix 500/500 atm
THICKNESS seems not to change the issue
struct PSInput
{
float4 Position : SV_POSITION;
};
float2 toScreenSpace(float4 vertex)
{
float2 WIN_SCALE = { 500.0f, 500.0f };
return float2(vertex.xy / vertex.w) * WIN_SCALE;
}
[maxvertexcount(7)]
void main(lineadj float4 vertices[4] : SV_POSITION, inout TriangleStream<PSInput> triStream)
{
float2 WIN_SCALE = { 500.0f, 500.0f };
float2 p0 = toScreenSpace(vertices[0]); // start of previous segment
float2 p1 = toScreenSpace(vertices[1]); // end of previous segment, start of current segment
float2 p2 = toScreenSpace(vertices[2]); // end of current segment, start of next segment
float2 p3 = toScreenSpace(vertices[3]); // end of next segment
// perform naive culling
float2 area = WIN_SCALE * 1.2;
if (p1.x < -area.x || p1.x > area.x)
return;
if (p1.y < -area.y || p1.y > area.y)
return;
if (p2.x < -area.x || p2.x > area.x)
return;
if (p2.y < -area.y || p2.y > area.y)
return;
float2 v0 = normalize(p1 - p0);
float2 v1 = normalize(p2 - p1);
float2 v2 = normalize(p3 - p2);
// determine the normal of each of the 3 segments (previous, current, next)
float2 n0 = { -v0.y, v0.x};
float2 n1 = { -v1.y, v1.x};
float2 n2 = { -v2.y, v2.x};
// determine miter lines by averaging the normals of the 2 segments
float2 miter_a = normalize(n0 + n1); // miter at start of current segment
float2 miter_b = normalize(n1 + n2); // miter at end of current segment
// determine the length of the miter by projecting it onto normal and then inverse it
float THICKNESS = 1;
float length_a = THICKNESS / dot(miter_a, n1);
float length_b = THICKNESS / dot(miter_b, n1);
float MITER_LIMIT = 0.75;
//float MITER_LIMIT = -1;
//float MITER_LIMIT = 0.1;
PSInput v;
float2 temp;
//// prevent excessively long miters at sharp corners
if (dot(v0, v1) < -MITER_LIMIT)
{
miter_a = n1;
length_a = THICKNESS;
// close the gap
if (dot(v0, n1) > 0)
{
temp = (p1 + THICKNESS * n0) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
temp = (p1 + THICKNESS * n1) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
v.Position = float4(p1 / WIN_SCALE, 1.0, 1.0);
triStream.Append(v);
//triStream.RestartStrip();
}
else
{
temp = (p1 - THICKNESS * n1) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
temp = (p1 - THICKNESS * n0) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
v.Position = float4(p1 / WIN_SCALE, 1.0, 1.0);
triStream.Append(v);
//triStream.RestartStrip();
}
}
if (dot(v1, v2) < -MITER_LIMIT)
{
miter_b = n1;
length_b = THICKNESS;
}
// generate the triangle strip
temp = (p1 + length_a * miter_a) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
temp = (p1 - length_a * miter_a) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
temp = (p2 + length_b * miter_b) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
temp = (p2 - length_b * miter_b) / WIN_SCALE;
v.Position = float4(temp, 1.0, 1.0);
triStream.Append(v);
//triStream.RestartStrip();
}
The thing is, that it won't output anything of the curve I give as points to it. It already works very well without thick lines, but I want to have that. I can add Noise to my Sinus signal and if I do that, (y-values increase by a little random %) suddenly the curve shows up and is thicker and seems to work, but I want it to be thick and showing without noise too.
If I up the frequency, without noise, point appear scattered-plot style, but not as a connected Sinus.
If I debug with the VS Graphics Debugger, I can see that in the frame with no Noise it says the Pixel-Shader-Stage is not run (?).
The frame with noise shows it run and working.
Maybe my porting is incorrect and I forget something, I am very new to programming shaders. Maybe I missunderstand the approach at all.
Any help appreciated.
I needed to set the default behavior of the culling to CullMode.None to get all the triangle drawn.

OpenCV + OpenGL Using solvePnP camera pose - object is offset from detected marker

I have a problem in my iOS application where i attempt to obtain a view matrix using solvePnP and render a 3d cube using modern OpenGL. While my code attempts to render a 3d cube directly on top of the detected marker, it seems to render with a certain offset from the marker (see video for example)
https://www.youtube.com/watch?v=HhP5Qr3YyGI&feature=youtu.be
(on the bottom right of the image you can see an opencv render of the homography around the tracker marker. the rest of the screen is an opengl render of the camera input frame and a 3d cube at location (0,0,0).
the cube rotates and translates correctly whenever i move the marker, though it is very telling that there is some difference in the scale of translations (IE, if i move my marker 5cm in the real world, it hardly moves by 1cm on screen)
these are what i believe to be the relevant parts of the code where the error could come from :
Extracting view matrix from homography :
AVCaptureDevice *deviceInput = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
AVCaptureDeviceFormat *format = deviceInput.activeFormat;
CMFormatDescriptionRef fDesc = format.formatDescription;
CGSize dim = CMVideoFormatDescriptionGetPresentationDimensions(fDesc, true, true);
const float cx = float(dim.width) / 2.0;
const float cy = float(dim.height) / 2.0;
const float HFOV = format.videoFieldOfView;
const float VFOV = ((HFOV)/cx)*cy;
const float fx = abs(float(dim.width) / (2 * tan(HFOV / 180 * float(M_PI) / 2)));
const float fy = abs(float(dim.height) / (2 * tan(VFOV / 180 * float(M_PI) / 2)));
Mat camIntrinsic = Mat::zeros(3, 3, CV_64F);
camIntrinsic.at<double>(0, 0) = fx;
camIntrinsic.at<double>(0, 2) = cx;
camIntrinsic.at<double>(1, 1) = fy;
camIntrinsic.at<double>(1, 2) = cy;
camIntrinsic.at<double>(2, 2) = 1.0;
std::vector<cv::Point3f> object3dPoints;
object3dPoints.push_back(cv::Point3f(-0.5f,-0.5f,0));
object3dPoints.push_back(cv::Point3f(+0.5f,-0.5f,0));
object3dPoints.push_back(cv::Point3f(+0.5f,+0.5f,0));
object3dPoints.push_back(cv::Point3f(-0.5f,+0.5f,0));
cv::Mat raux,taux;
cv::Mat Rvec, Tvec;
cv::solvePnP(object3dPoints, mNewImageBounds, camIntrinsic, Mat(),raux,taux); //mNewImageBounds are the 4 corner of the homography detected by perspectiveTransform (the green outline seen in the image)
raux.convertTo(Rvec,CV_32F);
taux.convertTo(Tvec ,CV_64F);
Mat Rot(3,3,CV_32FC1);
Rodrigues(Rvec, Rot);
// [R | t] matrix
Mat_<double> para = Mat_<double>::eye(4,4);
Rot.convertTo(para(cv::Rect(0,0,3,3)),CV_64F);
Tvec.copyTo(para(cv::Rect(3,0,1,3)));
Mat cvToGl = Mat::zeros(4, 4, CV_64F);
cvToGl.at<double>(0, 0) = 1.0f;
cvToGl.at<double>(1, 1) = -1.0f; // Invert the y axis
cvToGl.at<double>(2, 2) = -1.0f; // invert the z axis
cvToGl.at<double>(3, 3) = 1.0f;
para = cvToGl * para;
Mat_<double> modelview_matrix;
Mat(para.t()).copyTo(modelview_matrix); // transpose to col-major for OpenGL
glm::mat4 openGLViewMatrix;
for(int col = 0; col < modelview_matrix.cols; col++)
{
for(int row = 0; row < modelview_matrix.rows; row++)
{
openGLViewMatrix[col][row] = modelview_matrix.at<double>(col,row);
}
}
i made sure the camera intrinsic matrix contains correct values, the portion which converts the opencv Mat to an opengl view matrix i believe to be correct as the cube translates and rotates in the right directions.
once the view matrix is calculated, i use it to draw the cube as follows :
_projectionMatrix = glm::perspective<float>(radians(60.0f), fabs(view.bounds.size.width / view.bounds.size.height), 0.1f, 100.0f);
_cube_ModelMatrix = glm::translate(glm::vec3(0,0,0));
const mat4 MVP = _projectionMatrix * openGLViewMatrix * _cube_ModelMatrix;
glUniformMatrix4fv(glGetUniformLocation(_cube_program, "ModelMatrix"), 1, GL_FALSE, value_ptr(MVP));
glDrawElements(GL_TRIANGLES, 36, GL_UNSIGNED_INT, BUFFER_OFFSET(0));
Is anyone able to spot my error?
You should create perspective matrix as explained here: http://ksimek.github.io/2013/06/03/calibrated_cameras_in_opengl
Here is quick code:
const float fx = intrinsicParams(0, 0); // Focal length in x axis
const float fy = intrinsicParams(1, 1); // Focal length in y axis
const float cx = intrinsicParams(0, 2); // Primary point x
const float cy = intrinsicParams(1, 2); // Primary point y
projectionMatrix(0, 0) = 2.0f * fx;
projectionMatrix(0, 1) = 0.0f;
projectionMatrix(0, 2) = 0.0f;
projectionMatrix(0, 3) = 0.0f;
projectionMatrix(1, 0) = 0.0f;
projectionMatrix(1, 1) = 2.0f * fy;
projectionMatrix(1, 2) = 0.0f;
projectionMatrix(1, 3) = 0.0f;
projectionMatrix(2, 0) = 2.0f * cx - 1.0f;
projectionMatrix(2, 1) = 2.0f * cy - 1.0f;
projectionMatrix(2, 2) = -(far + near) / (far - near);
projectionMatrix(2, 3) = -1.0f;
projectionMatrix(3, 0) = 0.0f;
projectionMatrix(3, 1) = 0.0f;
projectionMatrix(3, 2) = -2.0f * far * near / (far - near);
projectionMatrix(3, 3) = 0.0f;
For more information about intrinsic matrix: http://ksimek.github.io/2013/08/13/intrinsic

touch object GLKIT or OpenGL any functions?

I am looking for a description to touch my OpenGL object or get an Event if I touch it.
Have the GLKit or OpenGL ES some functions to use?
Or I have to calculate the position of my Object and have to compare it with the coordination ob my Touch?
There is no built in one, but here is a ported version of the gluProject function that will put a point from your object in screen coordinates, so you can see if your touch is near that point:
GLKVector3 gluProject(GLKVector3 position,
GLKMatrix4 projMatrix,
GLKMatrix4 modelMatrix,
CGRect viewport
)
{
GLKVector4 in;
GLKVector4 out;
in = GLKVector4Make(position.x, position.y, position.z, 1.0);
out = GLKMatrix4MultiplyVector4(modelMatrix, in);
in = GLKMatrix4MultiplyVector4(projMatrix, out);
if (in.w == 0.0) NSLog(#"W = 0 in project function\n");
in.x /= in.w;
in.y /= in.w;
in.z /= in.w;
/* Map x, y and z to range 0-1 */
in.x = in.x * 0.5 + 0.5;
in.y = in.y * 0.5 + 0.5;
in.z = in.z * 0.5 + 0.5;
/* Map x,y to viewport */
in.x = in.x * (viewport.size.width) + viewport.origin.x;
in.y = in.y * (viewport.size.height) + viewport.origin.y;
return GLKVector3Make(in.x, in.y, in.z);
}
- (GLKVector2) getScreenCoordOfPoint {
GLKVector3 out = gluProject(self.point, modelMatrix, projMatrix, view.frame);
GLKVector2 point = GLKVector2Make(out.x, view.frame.size.height - out.y);
return point;
}

box2d with custom polygon crashes

I created a polygon which has 5 vertices, and all vertices are generated by VertexHelper.
Why does the program get SIGABRT at b2Assert(area > b2_epsilon) in ComputeCentroid() in b2PolygonShape.cpp?
The program runs well when I use shape.SetAsBox(.359375, 1.0) instead of shape.Set(vertices, count).
It seems that somethings wrong during calculating centroid when shape.Set() is used, but I don't know how to deal with this problem.
Here's the code:
b2BodyDef bodyDef;
bodyDef.type = b2_dynamicBody;
bodyDef.awake = NO;
bodyDef.position.Set(3.125, 3.125);
bodyDef.angle = -.785398163397;
spriteBody = world->CreateBody(&bodyDef);
spriteBody->SetUserData(sprite);
b2MassData massData = {2.0, b2Vec2(0.021875, 0.203125), 0.0};
spriteBody->SetMassData(&massData);
int32 count = 5;
b2Vec2 vertices[] = {
b2Vec2(-11.5f / PTM_RATIO, -16.0f / PTM_RATIO),
b2Vec2(-10.5f / PTM_RATIO, 15.0f / PTM_RATIO),
b2Vec2(10.5f / PTM_RATIO, 15.0f / PTM_RATIO),
b2Vec2(10.5f / PTM_RATIO, -5.0f / PTM_RATIO),
b2Vec2(-5.5f / PTM_RATIO, -16.0f / PTM_RATIO)
};
b2PolygonShape shape;
shape.Set(vertices, count);
b2FixtureDef fixtureDef;
fixtureDef.shape = &shape;
fixtureDef.density = 1.0f;
fixtureDef.friction = 0.2f;
fixtureDef.restitution = 0.7f;
spriteBody->CreateFixture(&fixtureDef);
It looks like you've wound your vertices the wrong way. I think they should be anticlockwise in box2d, at least by default.
You're assertion will be failing because the calculation for area will be returning a negative value, far less than b2_epsilon

Resources