Apply rotation around axis defined by touched point - ios

I have an object displayed using OpenGL ES on an iPad. The model is defined by vertices, normals and indexes to vertices. The origin of the model is 0,0,0. Using UIGestureRecognizer I can detect various gestures - two-fingered swipe horizontally for rotation about y, vertically for rotation about x. Two-fingered rotate gesture for rotation about y. Pan to move the model around. Pinch/zoom gesture to scale. I want the viewer to be able to manipulate the model to see (for example) the reverse of the model or the whole thing at once.
The basic strategy comes from Ray Wenderlich's tutorial but I have rewritten this in Swift.
I understand quaternions to be a vector and an angle. The vectors up, right and front represent the three axes:
front = GLKVector3Make(0.0, 0.0, 1.0)
right = GLKVector3Make(1.0, 0.0, 0.0)
up = GLKVector3Make(0.0, 1.0, 0.0)
so the quaternion apples a rotation around each of the three axes (though only one of dx, dy, dz has a value, decided by the gesture recognizer.)
func rotate(rotation : GLKVector3, multiplier : Float) {
let dx = rotation.x - rotationStart.x
let dy = rotation.y - rotationStart.y
let dz = rotation.z - rotationStart.z
rotationStart = GLKVector3Make(rotation.x, rotation.y, rotation.z)
rotationEnd = GLKQuaternionMultiply(GLKQuaternionMakeWithAngleAndVector3Axis(dx * multiplier, up), rotationEnd)
rotationEnd = GLKQuaternionMultiply(GLKQuaternionMakeWithAngleAndVector3Axis(dy * multiplier, right), rotationEnd)
rotationEnd = GLKQuaternionMultiply((GLKQuaternionMakeWithAngleAndVector3Axis(-dz, front)), rotationEnd)
state = .Rotation
}
Drawing uses the modelViewMatrix, calculated by the following function:
func modelViewMatrix() -> GLKMatrix4 {
var modelViewMatrix = GLKMatrix4Identity
// translation and zoom
modelViewMatrix = GLKMatrix4Translate(modelViewMatrix, translationEnd.x, translationEnd.y, -initialDepth);
// rotation
let quaternionMatrix = GLKMatrix4MakeWithQuaternion(rotationEnd)
modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, quaternionMatrix)
// scale
modelViewMatrix = GLKMatrix4Scale(modelViewMatrix, scaleEnd, scaleEnd, scaleEnd);
// rotation
return modelViewMatrix
}
And mostly this works. However everything is relative to the origin.
If the model is rotated then the pivot is always an axis passing through the origin - if zoomed in looking at the end of the model away from the origin and then rotating, the model can rapidly swing out of view. If the model is scaled then the origin is always the fixed point with the model growing larger or smaller - if the origin is off-screen and scale is reduced the model can disappear from view as it collapses toward the origin...
What should happen is that whatever the current view, the model rotates or scales relative to the current view. For a rotation around the y axis that would mean defining the y axis around which the rotation occurs as passing vertically through the middle of the current view. For a scale operation the fixed point of the model would be in the centre of the screen with the model shrinking toward or growing outward from that point.
I know that in 2D the solution is always to translate to the origin, apply rotation and then apply the inverse of the first translation. I don't see why this should be different in 3D, but I cannot find any example doing this with quaternions only matrices. I have tried to apply a translation and its inverse around the rotation but nothing has an effect.
So I tried to do this in the rotate function:
let xTranslation : Float = 300.0
let yTranslation : Float = 300.0
let translation = GLKMatrix4Translate(GLKMatrix4Identity, xTranslation, yTranslation, -initialDepth);
rotationEnd = GLKQuaternionMultiply(GLKQuaternionMakeWithMatrix4(translation) , rotationEnd)
rotationEnd = GLKQuaternionMultiply(GLKQuaternionMakeWithAngleAndVector3Axis(dx * multiplier, up), rotationEnd)
rotationEnd = GLKQuaternionMultiply(GLKQuaternionMakeWithAngleAndVector3Axis(dy * multiplier, right), rotationEnd)
rotationEnd = GLKQuaternionMultiply((GLKQuaternionMakeWithAngleAndVector3Axis(-dz, front)), rotationEnd)
// inverse translation
let inverseTranslation = GLKMatrix4Translate(GLKMatrix4Identity, -xTranslation, -yTranslation, -initialDepth);
rotationEnd = GLKQuaternionMultiply(GLKQuaternionMakeWithMatrix4(inverseTranslation) , rotationEnd)
The translation is 300,300 but there is no effect at all, it still pivots around where I know the origin to be. I've searched a long time for sample code and not found any.
The modelViewMatrix is applied in update() with:
effect?.transform.modelviewMatrix = modelViewMatrix
I could also cheat by adjusting all of the values in the model so that 0,0,0 falls at a central point - but that would still be a fixed origin and would be only marginally better.

The problem is in the last operation you made, you should swap the inverseTranslation with rotationEnd :
rotationEnd = GLKQuaternionMultiply(rotationEnd, GLKQuaternionMakeWithMatrix4(inverseTranslation))
And I think the partial rotation(dx, dy, dz) should follow the same rule.
In fact, if you want to change the pivot, this is how your matrix multiplication should be done:
modelMatrix = translationMatrix * rotationMatrix * inverse(translationMatrix)
and the result in homogeneous coordinates will be calculated as follows:
newPoint = translationMatrix * rotationMatrix * inverse(translationMatrix) * v4(x,y,z,1)
Example
This is a 2D test example that you can run in a playground.
let v4 = GLKVector4Make(1, 0, 0, 1) // Point A
let T = GLKMatrix4Translate(GLKMatrix4Identity, 1, 2, 0);
let rot = GLKMatrix4MakeWithQuaternion(GLKQuaternionMakeWithAngleAndVector3Axis(Float(M_PI)*0.5, GLKVector3Make(0, 0, 1))) //rotate by PI/2 around the z axis.
let invT = GLKMatrix4Translate(GLKMatrix4Identity, -1, -2, 0);
let partModelMat = GLKMatrix4Multiply(T, rot)
let modelMat = GLKMatrix4Multiply(partModelMat, invT) //The parameters were swapped in your code
//and the result would the rot matrix, since T*invT will be identity
var v4r = GLKMatrix4MultiplyVector4(modelMat, v4) //ModelMatrix multiplication with pointA
print(v4r.v) //(3,2,0,1)
//Step by step multiplication using the relation described above
v4r = GLKMatrix4MultiplyVector4(invT, v4)
v4r = GLKMatrix4MultiplyVector4(rot, v4r)
v4r = GLKMatrix4MultiplyVector4(T, v4r)
print(v4r.v) //(3,2,0,1)
As for the scale, if I understand correctly what you want, I would recommend to do it like it's done here: https://gamedev.stackexchange.com/questions/61473/combining-rotation-scaling-around-a-pivot-with-translation-into-a-matrix

Related

iOS revert camera projection

I'm trying to estimate my device position related to a QR code in space. I'm using ARKit and the Vision framework, both introduced in iOS11, but the answer to this question probably doesn't depend on them.
With the Vision framework, I'm able to get the rectangle that bounds a QR code in the camera frame. I'd like to match this rectangle to the device translation and rotation necessary to transform the QR code from a standard position.
For instance if I observe the frame:
* *
B
C
A
D
* *
while if I was 1m away from the QR code, centered on it, and assuming the QR code has a side of 10cm I'd see:
* *
A0 B0
D0 C0
* *
what has been my device transformation between those two frames? I understand that an exact result might not be possible, because maybe the observed QR code is slightly non planar and we're trying to estimate an affine transform on something that is not one perfectly.
I guess the sceneView.pointOfView?.camera?.projectionTransform is more helpful than the sceneView.pointOfView?.camera?.projectionTransform?.camera.projectionMatrix since the later already takes into account transform inferred from the ARKit that I'm not interested into for this problem.
How would I fill
func get transform(
qrCodeRectangle: VNBarcodeObservation,
cameraTransform: SCNMatrix4) {
// qrCodeRectangle.topLeft etc is the position in [0, 1] * [0, 1] of A0
// expected real world position of the QR code in a referential coordinate system
let a0 = SCNVector3(x: -0.05, y: 0.05, z: 1)
let b0 = SCNVector3(x: 0.05, y: 0.05, z: 1)
let c0 = SCNVector3(x: 0.05, y: -0.05, z: 1)
let d0 = SCNVector3(x: -0.05, y: -0.05, z: 1)
let A0, B0, C0, D0 = ?? // CGPoints representing position in
// camera frame for camera in 0, 0, 0 facing Z+
// then get transform from 0, 0, 0 to current position/rotation that sees
// a0, b0, c0, d0 through the camera as qrCodeRectangle
}
====Edit====
After trying number of things, I ended up going for camera pose estimation using openCV projection and perspective solver, solvePnP This gives me a rotation and translation that should represent the camera pose in the QR code referential. However when using those values and placing objects corresponding to the inverse transformation, where the QR code should be in the camera space, I get inaccurate shifted values, and I'm not able to get the rotation to work:
// some flavor of pseudo code below
func renderer(_ sender: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let currentFrame = sceneView.session.currentFrame, let pov = sceneView.pointOfView else { return }
let intrisics = currentFrame.camera.intrinsics
let QRCornerCoordinatesInQRRef = [(-0.05, -0.05, 0), (0.05, -0.05, 0), (-0.05, 0.05, 0), (0.05, 0.05, 0)]
// uses VNDetectBarcodesRequest to find a QR code and returns a bounding rectangle
guard let qr = findQRCode(in: currentFrame) else { return }
let imageSize = CGSize(
width: CVPixelBufferGetWidth(currentFrame.capturedImage),
height: CVPixelBufferGetHeight(currentFrame.capturedImage)
)
let observations = [
qr.bottomLeft,
qr.bottomRight,
qr.topLeft,
qr.topRight,
].map({ (imageSize.height * (1 - $0.y), imageSize.width * $0.x) })
// image and SceneKit coordinated are not the same
// replacing this by:
// (imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
// weirdly fixes an issue, see below
let rotation, translation = openCV.solvePnP(QRCornerCoordinatesInQRRef, observations, intrisics)
// calls openCV solvePnP and get the results
let positionInCameraRef = -rotation.inverted * translation
let node = SCNNode(geometry: someGeometry)
pov.addChildNode(node)
node.position = translation
node.orientation = rotation.asQuaternion
}
Here is the output:
where A, B, C, D are the QR code corners in the order they are passed to the program.
The predicted origin stays in place when the phone rotates, but it's shifted from where it should be. Surprisingly, if I shift the observations values, I'm able to correct this:
// (imageSize.height * (1 - $0.y), imageSize.width * $0.x)
// replaced by:
(imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
and now the predicted origin stays robustly in place. However I don't understand where the shift values come from.
Finally, I've tried to get an orientation fixed relatively to the QR code referential:
var n = SCNNode(geometry: redGeometry)
node.addChildNode(n)
n.position = SCNVector3(0.1, 0, 0)
n = SCNNode(geometry: blueGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0.1, 0)
n = SCNNode(geometry: greenGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0, 0.1)
The orientation is fine when I look at the QR code straight, but then it shifts by something that seems to be related to the phone rotation:
Outstanding questions I have are:
How do I solve the rotation?
where do the position shift values come from?
What simple relationship do rotation, translation, QRCornerCoordinatesInQRRef, observations, intrisics verify? Is it O ~ K^-1 * (R_3x2 | T) Q ? Because if so that's off by a few order of magnitude.
If that's helpful, here are a few numerical values:
Intrisics matrix
Mat 3x3
1090.318, 0.000, 618.661
0.000, 1090.318, 359.616
0.000, 0.000, 1.000
imageSize
1280.0, 720.0
screenSize
414.0, 736.0
==== Edit2 ====
I've noticed that the rotation works fine when the phone stays horizontally parallel to the QR code (ie the rotation matrix is [[a, 0, b], [0, 1, 0], [c, 0, d]]), no matter what the actual QR code orientation is:
Other rotation don't work.
Coordinate systems' correspondence
Take into consideration that Vision/CoreML coordinate system doesn't correspond to ARKit/SceneKit coordinate system. For details look at this post.
Rotation's direction
I suppose the problem is not in matrix. It's in vertices placement. For tracking 2D images you need to place ABCD vertices counter-clockwise (the starting point is A vertex located in imaginary origin x:0, y:0). I think Apple Documentation on VNRectangleObservation class (info about projected rectangular regions detected by an image analysis request) is vague. You placed your vertices in the same order as is in official documentation:
var bottomLeft: CGPoint
var bottomRight: CGPoint
var topLeft: CGPoint
var topRight: CGPoint
But they need to be placed the same way like positive rotation direction (about Z axis) occurs in Cartesian coordinates system:
World Coordinate Space in ARKit (as well as in SceneKit and Vision) always follows a right-handed convention (the positive Y axis points upward, the positive Z axis points toward the viewer and the positive X axis points toward the viewer's right), but is oriented based on your session's configuration. Camera works in Local Coordinate Space.
Rotation direction about any axis is positive (Counter-Clockwise) and negative (Clockwise). For tracking in ARKit and Vision it's critically important.
The order of rotation also makes sense. ARKit, as well as SceneKit, applies rotation relative to the node’s pivot property in the reverse order of the components: first roll (about Z axis), then yaw (about Y axis), then pitch (about X axis). So the rotation order is ZYX.
Math (Trig.):
Notes: the bottom is l (the QR code length), the left angle is k, and the top angle is i (the camera)

Using CATransform3DRotate with perspective: how to correct the 2D size increase?

I'm trying to create a paper folding effect in Swift using CALayers and CATransform3DRotate. There are some libraries out there, but those are pretty outdated and don't fit my needs (they don't have symmetric folds, for example).
My content view controller will squeeze to the right half side of the screen, revealing the menu at the left side.
Everything went well, until I applied perspective: then the dimensions I calculate are not correct anymore.
To explain the problem, I created a demo to show you what I'm doing.
This the content view controller with three squares. I will use three folds, so each square will be on a separate fold.
The even folds will get anchor point (0, 0.5) and the odd folds will get anchor point (1, 0.5), plus they'll receive a shadow.
When fully folded, the content view will be half of the screen's width.
On an iPhone 7, each fold/plane will be 125 points unfolded and 62.5 points fully folded when looked at.
To calculate the rotation needed to achieve this 62.5 points width, we can use a trigonometric function. To illustrate, look at this top-down view:
We know the original plane size (125) and the 2D width (62.5), so we can calculate the angle α using arccos:
let angle = acos(width / originalWidth)
The result is 1.04719755 rad or 60 degrees.
When using this formula with CATransform3DRotate, I get the correct result:
Now for the problem: when I add perspective, my calculation isn't correct anymore. The planes are bigger. Probably because of the now different projection.
You can see the planes are now overlapping and being clipped.
I reconstructed the desired result on the right by playing with the angle, but the correction needed is not consistent, unfortunately.
Here's the code I use. It works perfectly without perspective.
// Loop layers
for i in 0..<self.layers.count {
// Get layer
let layer = self.layers[i]
// Get dimensions
let width = self.frame.size.width / CGFloat(self.numberOfFolds)
let originalWidth = self.sourceView.frame.size.width / CGFloat(self.numberOfFolds)
// Calculate angle
let angle = acos(width / originalWidth)
// Set transform
layer.transform = CATransform3DIdentity
layer.transform.m34 = 1.0 / -500
layer.transform = CATransform3DRotate(layer.transform, angle * (i % 2 == 0 ? -1 : 1), 0, 1, 0)
// Update position
if i % 2 == 0 {
layer.position = CGPoint(x: (width * CGFloat(i)), y: layer.position.y)
} else {
layer.position = CGPoint(x: (width * CGFloat(i + 1)), y: layer.position.y)
}
}
So my question is: how do I achieve the desired result? Do I need to correct the angle, or should I calculate the projected/2D width differently?
Thanks in advance! :)

Why is my shape distorted on rotation about the z axis?

I just started learning metal and can best show you my frustration with the following series of screenshots. From top to bottom we have
(1) My model where the model matrix is the identity matrix
(2) My model rotated 60 deg about the x axis with orthogonal projection
(3) My model rotated 60 deg about the y axis with orthogonal projection
(4) My model rotated 60 deg about the z axis
So I use the following function for conversion into normalized device coordinates:
- (CGPoint)normalizedDevicePointForViewPoint:(CGPoint)point
{
CGPoint p = [self convertPoint:point toCoordinateSpace:self.window.screen.fixedCoordinateSpace];
CGFloat halfWidth = CGRectGetMidX(self.window.screen.bounds);
CGFloat halfHeight = CGRectGetMidY(self.window.screen.bounds);
CGFloat px = ( p.x - halfWidth ) / halfWidth;
CGFloat py = ( p.y - halfHeight ) / halfHeight;
return CGPointMake(px, -py);
}
The following rotates and orthogonally projects the model:
- (matrix_float4x4)zRotation
{
self.rotationZ = M_PI / 3;
const vector_float3 zAxis = { 0, 0, 1 };
const matrix_float4x4 zRot = matrix_float4x4_rotation(zAxis, self.rotationZ);
const matrix_float4x4 modelMatrix = zRot;
return matrix_multiply( matrix_float4x4_orthogonal_projection_on_z_plane(), modelMatrix );
}
As you can see when I use the exact same method for rotating about the other two axes, it looks fine-not distorted. What am I doing wrong? Is there some sort of scaling/aspect ratio thing I should be setting somewhere? What things could it be? I've been staring at this for an embarrassingly long period of time so any help/ideas that can lead me in the right direction are much appreciated. Thank you in advance.
There's nothing wrong with your rotation or projection matrices. The visual oddity arises from the fact that you move your vertices into NDC space prior to rotation. A rectangle doesn't preserve its aspect ratio when rotating in NDC space, because the mapping from NDC back to screen coordinates is not 1:1.
I would recommend not working in NDC until the very end of the vertex pipeline (i.e., pass vertices into your vertex function in "world" space, and out to the rasterizer as NDC). You can do this with a classic construction of the orthographic projection matrix that scales and biases the vertices, correctly accounting for the non-square aspect ratio of window coordinates.

How can I track a point on a texture in OpenGL ES1?

In my iOS application I have a texture applied to a sphere rendered in OpenGLES1. The sphere can be rotated by the user. How can I track where a given point on the texture is in 2D space at any given time?
For example, given point (200, 200) on a texture that's 1000px x 1000px, I'd like to place a UIButton on top of my OpenGL view that tracks the point as the sphere is manipulated.
What's the best way to do this?
On my first attempt, I tried to use a color-picking technique where I have a separate sphere in an off-screen framebuffer that uses a black texture with a red square at point (200, 200). Then, I used glReadPixels() to track the position of the red square and I moved my button accordingly. Unfortunately, grabbing all the pixel data and iterating it 60 times a second just isn't possible for obvious performance reasons. I tried a number of ways to optimize this hack (eg: iterating only the red pixels, iterating every 4th red pixel, etc), but it just didn't prove to be reliable.
I'm an OpenGL noob, so I'd appreciate any guidance. Is there a better solution? Thanks!
I think it's easier to keep track of where your ball is instead of searching for it with pixels. Then just have a couple of functions to translate your ball's coordinates to your view's coordinates (and back), then set your subview's center to the translated coordinates.
CGPoint translatePointFromGLCoordinatesToUIView(CGPoint coordinates, UIView *myGLView){
//if your drawing coordinates were between (horizontal {-1.0 -> 1.0} vertical {-1 -> 1})
CGFloat leftMostGLCoord = -1;
CGFloat rightMostGLCoord = 1;
CGFloat bottomMostGLCoord = -1;
CGFloat topMostGLCoord = 1;
CGPoint scale;
scale.x = (rightMostGLCoord - leftMostGLCoord) / myGLView.bounds.size.width;
scale.y = (topMostGLCoord - bottomMostGLCoord) / myGLView.bounds.size.height;
coordinates.x -= leftMostGLCoord;
coordinates.y -= bottomMostGLCoord;
CGPoint translatedPoint;
translatedPoint.x = coordinates.x / scale.x;
translatedPoint.y =coordinates.y / scale.y;
//flip y for iOS coordinates
translatedPoint.y = myGLView.bounds.size.height - translatedPoint.y;
return translatedPoint;
}
CGPoint translatePointFromUIViewToGLCoordinates(CGPoint pointInView, UIView *myGLView){
//if your drawing coordinates were between (horizontal {-1.0 -> 1.0} vertical {-1 -> 1})
CGFloat leftMostGLCoord = -1;
CGFloat rightMostGLCoord = 1;
CGFloat bottomMostGLCoord = -1;
CGFloat topMostGLCoord = 1;
CGPoint scale;
scale.x = (rightMostGLCoord - leftMostGLCoord) / myGLView.bounds.size.width;
scale.y = (topMostGLCoord - bottomMostGLCoord) / myGLView.bounds.size.height;
//flip y for iOS coordinates
pointInView.y = myGLView.bounds.size.height - pointInView.y;
CGPoint translatedPoint;
translatedPoint.x = leftMostGLCoord + (pointInView.x * scale.x);
translatedPoint.y = bottomMostGLCoord + (pointInView.y * scale.y);
return translatedPoint;
}
In my app I choose to use the iOS coordinate system for my drawing too. I just apply a projection matrix to my whole glkView the reconciles the coordinate system.
static GLKMatrix4 GLKMatrix4MakeIOSCoordsWithSize(CGSize screenSize){
GLKMatrix4 matrix4 = GLKMatrix4MakeScale(
2.0 / screenSize.width,
-2.0 / screenSize.height,
1.0);
matrix4 = GLKMatrix4Translate(matrix4,-screenSize.width / 2.0, -screenSize.height / 2.0, 0);
return matrix4;
}
This way you don't have to translate anything.

OpenCV: rotation/translation vector to OpenGL modelview matrix

I'm trying to use OpenCV to do some basic augmented reality. The way I'm going about it is using findChessboardCorners to get a set of points from a camera image. Then, I create a 3D quad along the z = 0 plane and use solvePnP to get a homography between the imaged points and the planar points. From that, I figure I should be able to set up a modelview matrix which will allow me to render a cube with the right pose on top of the image.
The documentation for solvePnP says that it outputs a rotation vector "that (together with [the translation vector] ) brings points from the model coordinate system to the camera coordinate system." I think that's the opposite of what I want; since my quad is on the plane z = 0, I want a a modelview matrix which will transform that quad to the appropriate 3D plane.
I thought that by performing the opposite rotations and translations in the opposite order I could calculate the correct modelview matrix, but that seems not to work. While the rendered object (a cube) does move with the camera image and seems to be roughly correct translationally, the rotation just doesn't work at all; it on multiple axes when it should only be rotating on one, and sometimes in the wrong direction. Here's what I'm doing so far:
std::vector<Point2f> corners;
bool found = findChessboardCorners(*_imageBuffer, cv::Size(5,4), corners,
CV_CALIB_CB_FILTER_QUADS |
CV_CALIB_CB_FAST_CHECK);
if(found)
{
drawChessboardCorners(*_imageBuffer, cv::Size(6, 5), corners, found);
std::vector<double> distortionCoefficients(5); // camera distortion
distortionCoefficients[0] = 0.070969;
distortionCoefficients[1] = 0.777647;
distortionCoefficients[2] = -0.009131;
distortionCoefficients[3] = -0.013867;
distortionCoefficients[4] = -5.141519;
// Since the image was resized, we need to scale the found corner points
float sw = _width / SMALL_WIDTH;
float sh = _height / SMALL_HEIGHT;
std::vector<Point2f> board_verts;
board_verts.push_back(Point2f(corners[0].x * sw, corners[0].y * sh));
board_verts.push_back(Point2f(corners[15].x * sw, corners[15].y * sh));
board_verts.push_back(Point2f(corners[19].x * sw, corners[19].y * sh));
board_verts.push_back(Point2f(corners[4].x * sw, corners[4].y * sh));
Mat boardMat(board_verts);
std::vector<Point3f> square_verts;
square_verts.push_back(Point3f(-1, 1, 0));
square_verts.push_back(Point3f(-1, -1, 0));
square_verts.push_back(Point3f(1, -1, 0));
square_verts.push_back(Point3f(1, 1, 0));
Mat squareMat(square_verts);
// Transform the camera's intrinsic parameters into an OpenGL camera matrix
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
// Camera parameters
double f_x = 786.42938232; // Focal length in x axis
double f_y = 786.42938232; // Focal length in y axis (usually the same?)
double c_x = 217.01358032; // Camera primary point x
double c_y = 311.25384521; // Camera primary point y
cv::Mat cameraMatrix(3,3,CV_32FC1);
cameraMatrix.at<float>(0,0) = f_x;
cameraMatrix.at<float>(0,1) = 0.0;
cameraMatrix.at<float>(0,2) = c_x;
cameraMatrix.at<float>(1,0) = 0.0;
cameraMatrix.at<float>(1,1) = f_y;
cameraMatrix.at<float>(1,2) = c_y;
cameraMatrix.at<float>(2,0) = 0.0;
cameraMatrix.at<float>(2,1) = 0.0;
cameraMatrix.at<float>(2,2) = 1.0;
Mat rvec(3, 1, CV_32F), tvec(3, 1, CV_32F);
solvePnP(squareMat, boardMat, cameraMatrix, distortionCoefficients,
rvec, tvec);
_rv[0] = rvec.at<double>(0, 0);
_rv[1] = rvec.at<double>(1, 0);
_rv[2] = rvec.at<double>(2, 0);
_tv[0] = tvec.at<double>(0, 0);
_tv[1] = tvec.at<double>(1, 0);
_tv[2] = tvec.at<double>(2, 0);
}
Then in the drawing code...
GLKMatrix4 modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, 0.0f);
modelViewMatrix = GLKMatrix4Translate(modelViewMatrix, -tv[1], -tv[0], -tv[2]);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, -rv[0], 1.0f, 0.0f, 0.0f);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, -rv[1], 0.0f, 1.0f, 0.0f);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, -rv[2], 0.0f, 0.0f, 1.0f);
The vertices I'm rendering create a cube of unit length around the origin (i.e. from -0.5 to 0.5 along each edge.) I know with OpenGL translation functions performed transformations in "reverse order," so the above should rotate the cube along the z, y, and then x axes, and then translate it. However, it seems like it's being translated first and then rotated, so perhaps Apple's GLKMatrix4 works differently?
This question seems very similar to mine, and in particular coder9's answer seems like it might be more or less what I'm looking for. However, I tried it and compared the results to my method, and the matrices I arrived at in both cases were the same. I feel like that answer is right, but that I'm missing some crucial detail.
You have to make sure the axis are facing the correct direction. Especially, the y and z axis are facing different directions in OpenGL and OpenCV to ensure the x-y-z basis is direct. You can find some information and code (with an iPad camera) in this blog post.
-- Edit --
Ah ok. Unfortunately, I used these resources to do it the other way round (opengl ---> opencv) to test some algorithms. My main issue was that the row order of the images was inverted between OpenGL and OpenCV (maybe this helps).
When simulating cameras, I came across the same projection matrices that can be found here and in the generalized projection matrix paper. This paper quoted in the comments of the blog post also shows some link between computer vision and OpenGL projections.
I'm not an IOS programmer, so this answer might be misleading!
If the problem is not in the order of applying the rotations and the translation, then suggest using a simpler and more commonly used coordinate system.
The points in the corners vector have the origin (0,0) at the top left corner of the image and the y axis is towards the bottom of the image. Often from math we are used to think of the coordinate system with the origin at the center and y axis towards the top of the image. From the coordinates you're pushing into board_verts I'm guessing you're making the same mistake. If that's the case, it's easy to transform the positions of the corners by something like this:
for (i=0;i<corners.size();i++) {
corners[i].x -= width/2;
corners[i].y = -corners[i].y + height/2;
}
then you call solvePnP(). Debugging this is not that difficult, just print the positions of the four corners and the estimated R and T, and see if they make sense. Then you can proceed to the OpenGL step. Please let me know how it goes.

Resources