Related
I'm trying to draw multiple cubes at an isometric camera angle. Here's the code that draws one. (OpenGL ES 2.0 with GLKit on iOS).
float startZ = -4.0f;
// position
GLKMatrix4 modelViewMatrix = GLKMatrix4Identity;
modelViewMatrix = GLKMatrix4Translate(modelViewMatrix, location.x, location.y, location.z + startZ);
// isometric camera angle
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, GLKMathDegreesToRadians(45), 1.0, 0, 0);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, GLKMathDegreesToRadians(45), 0.0, 1.0, 0);
self.effect.transform.modelviewMatrix = modelViewMatrix;
[self.effect prepareToDraw];
glDrawArrays(GL_TRIANGLES, 0, 36);
The problem is that it is translating first, then rotating, which means with more than one box, they do not line up (they look like a chain of diamonds. Each one is in position and rotated so the corners overlap).
I've tried switching the order so the rotation is before the translation, but they don't show up at all. My vertex array is bound to a unit cube centered around the origin.
I really don't understand how to control the camera separate from the object. I screwed around with the projection matrix for a while without getting it. As far as I understand, the camera is supposed to be controlled with the modelViewMatrix, right? (The "View" part).
Your 'camera' transform (modelview) seems correct, however it looks like you're using a perspective projection - if you want isometric you will need to change your projection matrix.
It looks like you are applying the camera rotation to each object as you draw it. Instead, simulate a 2-deep matrix stack if your use-case is this simple, so you just have your camera matrix and each cube's matrix.
Set your projection and camera matrices - keep a reference to your camera matrix.
For each cube generate the individual cube transformation matrix (which should probably consist of translation only - no rotation so the cubes remain axis aligned - I think this is what you're going for).
Backwards-multiply your cube matrix by the camera matrix and use that as the modelview matrix.
Note that the camera matrix will remain unchanged for each cube you render, but the modelview matrix will incorporate both the cube's individual transformation matrix and the camera matrix into a single modelview matrix. This is equivalent to the old matrix stack methods glPushMatrix and glPopMatrix (not available in GLES 2.0). If you need more complex object hierarchies (where the cubes have child-objects in their 'local' coordinate space) then you should probably implement your own full matrix stack, instead of the 2-deep equivalent discussed above.
For reference, here's an article that helped me understand it. It's a little mathy, but does a good job explaining things intuitively.
http://db-in.com/blog/2011/04/cameras-on-opengl-es-2-x/
I ended up keeping the perspective projection, because I don't want true isometric. The key was to do them in the right order, because moving the camera is the inverse of moving the object. See the comments, and the article. Working code:
// the one you want to happen first is multiplied LAST
// camRotate * camScale * camTranslate * objTranslate * objScale * objRotate;
// TODO cache the camera matrix
// the camera angle remains the same for all objects
GLKMatrix4 camRotate = GLKMatrix4MakeRotation(GLKMathDegreesToRadians(45), 1, 0, 0);
camRotate = GLKMatrix4Rotate(camRotate, GLKMathDegreesToRadians(45), 0, 1, 0);
GLKMatrix4 camTranslate = GLKMatrix4MakeTranslation(4, -5, -4.0);
GLKMatrix4 objTranslate = GLKMatrix4MakeTranslation(location.x, location.y, location.z);
GLKMatrix4 modelViewMatrix = GLKMatrix4Multiply(camRotate, camTranslate);
modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, objTranslate);
self.effect.transform.modelviewMatrix = modelViewMatrix;
[self.effect prepareToDraw];
glDrawArrays(GL_TRIANGLES, 0, 36);
I'd like to take the perspective transform matrix returned from OpenCV's findHomography function and convert it (either in C++ or Objective-C) to iOS' CATransform3D. I'd like them to be as close as possible in terms of accurately reproducing the "warp" effect on the Core Graphics side. Example code would really be appreciated!
From iOS' CATransform3D.h:
/* Homogeneous three-dimensional transforms. */
struct CATransform3D
{
CGFloat m11, m12, m13, m14;
CGFloat m21, m22, m23, m24;
CGFloat m31, m32, m33, m34;
CGFloat m41, m42, m43, m44;
};
Similar questions:
Apply homography matrix using Core Graphics
convert an opencv affine matrix to CGAffineTransform
Disclaimer
I have never tried this so take it with a grain of salt.
CATRansform3D is a 4x4 matrix which operates on a 3 dimensional homogeneous vector (4x1) to produce another vector of the same type. I am assuming that when rendered, objects described by a 4x1 vector have each element divided by the 4th element and the 3rd element is used only to determine which objects appear on top of which. Assuming this is correct...
Reasoning
The 3x3 matrix returned by findHomography operates on a 2 dimensional homogeneous vector. That process can be thought of in 4 steps
The first column of the homography is multiplied by x
The second column of the homography is multiplied by y
The third column of the homography is multiplied by 1
the resulting 1st and 2nd vector elements are divided by the 3rd
You need this process to be replicated in a 4x4 vector in which I am assuming the 3rd element in the resulting vector is meaningless for your purposes.
Solution
Construct your matrix like this (H is your homography matrix)
[H(0,0), H(0,1), 0, H(0,2),
H(1,0), H(1,1), 0, H(1,2),
0, 0, 1, 0
H(2,0), H(2,1), 0, H(2,2)]
This clearly satisfies 1,2 and 3. 4 is satisfied because the homogeneous element is always the last one. That is why the "homogeneous row" if you will had to get bumped down one line. The 1 on the 3rd row is to let the z component of the vector pass through unmolested.
All of the above is done in row major notation (like openCV) to try to keep things from being confusing. You can look at Tommy's answer to see how the conversion to column major looks (you basically just transpose it). Note however that at the moment Tommy and I disagree about how to construct the matrix.
From my reading of the documentation, m11 in CATransform3D is equivalent to a in CGAffineTransform, m12 is equivalent to b and so on.
As per your comment below, I understand the matrix OpenCV returns to be 3x3 (which, in retrospect, is the size you'd expect). So you'd fill in the other elements with those equivalent to the identity matrix. As per Hammer's answer, you want to preserve the portion that deals with the (usually implicit) homogenous coordinate in its place while padding everything else with the identity.
[aside: my original answer was wrong. I've edited it to be correct since I've posted code and Hammer hasn't. This post is marked as community wiki to reflect that it's in no sense solely my answer]
So I think you'd want:
CATransform3D MatToTransform(Mat cvTransform)
{
CATransform3D transform;
transform.m11 = cvTransform.at<float>(0, 0);
transform.m12 = cvTransform.at<float>(1, 0);
transform.m13 = 0.0f;
transform.m14 = cvTransform.at<float>(2, 0);
transform.m21 = cvTransform.at<float>(0, 1);
transform.m22 = cvTransform.at<float>(1, 1);
transform.m23 = 0.0f;
transform.m24 = cvTransform.at<float>(2, 1);
transform.m31 = 0.0f;
transform.m32 = 0.0f;
transform.m33 = 1.0f;
transform.m34 = 0.0f;
transform.m41 = cvTransform.at<float>(0, 2);
transform.m42 = cvTransform.at<float>(1, 2);
transform.m43 = 0.0f;
transform.m44 = cvTransform.at<float>(2, 2);
return transform;
}
Or use cvGetReal1D if you're keeping C++ out of it.
Tommy answer worked for me, but I needed to use double, instead of float. This is also shortened version of the code:
CATransform3D MatToCATransform3D(cv::Mat H) {
return {
H.at<double>(0, 0), H.at<double>(1, 0), 0.0, H.at<double>(2, 0),
H.at<double>(0, 1), H.at<double>(1, 1), 0.0, H.at<double>(2, 1),
0.0, 0.0, 1.0, 0.0,
H.at<double>(0, 2), H.at<double>(1, 2), 0.0f, H.at<double>(2, 2)
};
}
I'm trying to use OpenCV to do some basic augmented reality. The way I'm going about it is using findChessboardCorners to get a set of points from a camera image. Then, I create a 3D quad along the z = 0 plane and use solvePnP to get a homography between the imaged points and the planar points. From that, I figure I should be able to set up a modelview matrix which will allow me to render a cube with the right pose on top of the image.
The documentation for solvePnP says that it outputs a rotation vector "that (together with [the translation vector] ) brings points from the model coordinate system to the camera coordinate system." I think that's the opposite of what I want; since my quad is on the plane z = 0, I want a a modelview matrix which will transform that quad to the appropriate 3D plane.
I thought that by performing the opposite rotations and translations in the opposite order I could calculate the correct modelview matrix, but that seems not to work. While the rendered object (a cube) does move with the camera image and seems to be roughly correct translationally, the rotation just doesn't work at all; it on multiple axes when it should only be rotating on one, and sometimes in the wrong direction. Here's what I'm doing so far:
std::vector<Point2f> corners;
bool found = findChessboardCorners(*_imageBuffer, cv::Size(5,4), corners,
CV_CALIB_CB_FILTER_QUADS |
CV_CALIB_CB_FAST_CHECK);
if(found)
{
drawChessboardCorners(*_imageBuffer, cv::Size(6, 5), corners, found);
std::vector<double> distortionCoefficients(5); // camera distortion
distortionCoefficients[0] = 0.070969;
distortionCoefficients[1] = 0.777647;
distortionCoefficients[2] = -0.009131;
distortionCoefficients[3] = -0.013867;
distortionCoefficients[4] = -5.141519;
// Since the image was resized, we need to scale the found corner points
float sw = _width / SMALL_WIDTH;
float sh = _height / SMALL_HEIGHT;
std::vector<Point2f> board_verts;
board_verts.push_back(Point2f(corners[0].x * sw, corners[0].y * sh));
board_verts.push_back(Point2f(corners[15].x * sw, corners[15].y * sh));
board_verts.push_back(Point2f(corners[19].x * sw, corners[19].y * sh));
board_verts.push_back(Point2f(corners[4].x * sw, corners[4].y * sh));
Mat boardMat(board_verts);
std::vector<Point3f> square_verts;
square_verts.push_back(Point3f(-1, 1, 0));
square_verts.push_back(Point3f(-1, -1, 0));
square_verts.push_back(Point3f(1, -1, 0));
square_verts.push_back(Point3f(1, 1, 0));
Mat squareMat(square_verts);
// Transform the camera's intrinsic parameters into an OpenGL camera matrix
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
// Camera parameters
double f_x = 786.42938232; // Focal length in x axis
double f_y = 786.42938232; // Focal length in y axis (usually the same?)
double c_x = 217.01358032; // Camera primary point x
double c_y = 311.25384521; // Camera primary point y
cv::Mat cameraMatrix(3,3,CV_32FC1);
cameraMatrix.at<float>(0,0) = f_x;
cameraMatrix.at<float>(0,1) = 0.0;
cameraMatrix.at<float>(0,2) = c_x;
cameraMatrix.at<float>(1,0) = 0.0;
cameraMatrix.at<float>(1,1) = f_y;
cameraMatrix.at<float>(1,2) = c_y;
cameraMatrix.at<float>(2,0) = 0.0;
cameraMatrix.at<float>(2,1) = 0.0;
cameraMatrix.at<float>(2,2) = 1.0;
Mat rvec(3, 1, CV_32F), tvec(3, 1, CV_32F);
solvePnP(squareMat, boardMat, cameraMatrix, distortionCoefficients,
rvec, tvec);
_rv[0] = rvec.at<double>(0, 0);
_rv[1] = rvec.at<double>(1, 0);
_rv[2] = rvec.at<double>(2, 0);
_tv[0] = tvec.at<double>(0, 0);
_tv[1] = tvec.at<double>(1, 0);
_tv[2] = tvec.at<double>(2, 0);
}
Then in the drawing code...
GLKMatrix4 modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, 0.0f);
modelViewMatrix = GLKMatrix4Translate(modelViewMatrix, -tv[1], -tv[0], -tv[2]);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, -rv[0], 1.0f, 0.0f, 0.0f);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, -rv[1], 0.0f, 1.0f, 0.0f);
modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, -rv[2], 0.0f, 0.0f, 1.0f);
The vertices I'm rendering create a cube of unit length around the origin (i.e. from -0.5 to 0.5 along each edge.) I know with OpenGL translation functions performed transformations in "reverse order," so the above should rotate the cube along the z, y, and then x axes, and then translate it. However, it seems like it's being translated first and then rotated, so perhaps Apple's GLKMatrix4 works differently?
This question seems very similar to mine, and in particular coder9's answer seems like it might be more or less what I'm looking for. However, I tried it and compared the results to my method, and the matrices I arrived at in both cases were the same. I feel like that answer is right, but that I'm missing some crucial detail.
You have to make sure the axis are facing the correct direction. Especially, the y and z axis are facing different directions in OpenGL and OpenCV to ensure the x-y-z basis is direct. You can find some information and code (with an iPad camera) in this blog post.
-- Edit --
Ah ok. Unfortunately, I used these resources to do it the other way round (opengl ---> opencv) to test some algorithms. My main issue was that the row order of the images was inverted between OpenGL and OpenCV (maybe this helps).
When simulating cameras, I came across the same projection matrices that can be found here and in the generalized projection matrix paper. This paper quoted in the comments of the blog post also shows some link between computer vision and OpenGL projections.
I'm not an IOS programmer, so this answer might be misleading!
If the problem is not in the order of applying the rotations and the translation, then suggest using a simpler and more commonly used coordinate system.
The points in the corners vector have the origin (0,0) at the top left corner of the image and the y axis is towards the bottom of the image. Often from math we are used to think of the coordinate system with the origin at the center and y axis towards the top of the image. From the coordinates you're pushing into board_verts I'm guessing you're making the same mistake. If that's the case, it's easy to transform the positions of the corners by something like this:
for (i=0;i<corners.size();i++) {
corners[i].x -= width/2;
corners[i].y = -corners[i].y + height/2;
}
then you call solvePnP(). Debugging this is not that difficult, just print the positions of the four corners and the estimated R and T, and see if they make sense. Then you can proceed to the OpenGL step. Please let me know how it goes.
How can I make a Core Graphics affine transform for rotation around a point x,y of angle a, using only a single call to CGAffineTransformMake() plus math.h trig functions such as sin(), cos(), etc., and no other CG calls.
Other answers here seem to be about using multiple stacked transforms or multi-step transforms to move, rotate and move, using multiple Core Graphics calls. Those answers do not meet my specific requirements.
A rotation of angle a around the point (x,y) corresponds to the affine transformation:
CGAffineTransform transform = CGAffineTransformMake(cos(a),sin(a),-sin(a),cos(a),x-x*cos(a)+y*sin(a),y-x*sin(a)-y*cos(a));
You may need to plug in -a instead of a depending on whether you want the rotation to be clockwise or counterclockwise. Also, you may need to plug in -y instead of y depending on whether or not your coordinate system is upside down.
Also, you can accomplish precisely the same thing in three lines of code using:
CGAffineTransform transform = CGAffineTransformMakeTranslation(x, y);
transform = CGAffineTransformRotate(transform, a);
transform = CGAffineTransformTranslate(transform,-x,-y);
If you were applying this to a view, you could also simply use a rotation transform via CGAffineTransformMakeRotation(a), provided you set the view's layer's anchorPoint property to reflect the point you want to rotate around. However, is sounds like you aren't interested in applying this to a view.
Finally, if you are applying this to a non-Euclidean 2D space, you may not want an affine transformation at all. Affine transformations are isometries of Euclidean space, meaning that they preserve the standard Euclidean distance, as well as angles. If your space is not Euclidean, then the transformation you want may not actually be affine, or if it is affine, the matrix for the rotation might not be as simple as what I wrote above with sin and cos. For instance, if you were in a hyperbolic space, you might need to use the hyperbolic trig functions sinh and cosh, along with different + and - signs in the formula.
P.S. I also wanted to remind anyone reading this far that "affine" is pronounced with a short "a" as in "ask", not a long "a" as in "able". I have even heard Apple employees mispronouncing it in their WWDC talks.
for Swift 4
print(x, y) // where x,y is the point to rotate around
let degrees = 45.0
let transform = CGAffineTransform(translationX: x, y: y)
.rotated(by: degrees * .pi / 180)
.translatedBy(x: -x, y: -y)
For those like me, that are struggling in search of a complete solution to rotate an image and scale it properly, in order to fill the containing frame, after a couple of hours this is the most complete and flawless solution that I have obtained.
The trick here is to translate the reference point, before any trasformation involved (both scale and rotation). After that, you have to concatenate the two transform in order to obtain a complete affine transform.
I have packed the whole solution in a CIFilter subclass that you can gist here.
Following the relevant part of code:
CGFloat a = _inputDegree.floatValue;
CGFloat x = _inputImage.extent.size.width/2.0;
CGFloat y = _inputImage.extent.size.height/2.0;
CGFloat scale = [self calculateScaleForAngle:GLKMathRadiansToDegrees(a)];
CGAffineTransform transform = CGAffineTransformMakeTranslation(x, y);
transform = CGAffineTransformRotate(transform, a);
transform = CGAffineTransformTranslate(transform,-x,-y);
CGAffineTransform transform2 = CGAffineTransformMakeTranslation(x, y);
transform2 = CGAffineTransformScale(transform2, scale, scale);
transform2 = CGAffineTransformTranslate(transform2,-x,-y);
CGAffineTransform concate = CGAffineTransformConcat(transform2, transform);
Here's some convenience methods for rotating about an anchor point:
extension CGAffineTransform {
init(rotationAngle: CGFloat, anchor: CGPoint) {
self.init(
a: cos(rotationAngle),
b: sin(rotationAngle),
c: -sin(rotationAngle),
d: cos(rotationAngle),
tx: anchor.x - anchor.x * cos(rotationAngle) + anchor.y * sin(rotationAngle),
ty: anchor.y - anchor.x * sin(rotationAngle) - anchor.y * cos(rotationAngle)
)
}
func rotated(by angle: CGFloat, anchor: CGPoint) -> Self {
let transform = Self(rotationAngle: angle, anchor: anchor)
return self.concatenating(transform)
}
}
Use the view's layer and anchor point. e.g.
view.layer.anchorPoint = CGPoint(x:0,y:1.0)
If I want a orthograpic view of the region (-10,-1),(-2,-1),(-2,-7),(-10,-7) how do I define the view and projection matrices? (I set the world matrix to identity). I tried this but it did not work:
worldMatrix = Matrix.Identity;
projectionMatrix = Matrix.CreateOrthographicOffCenter(-10,-2,-7,-1,-1.0f,100.0f);
viewMatrix = Matrix.CreateTranslation(1, -1, 0) * Matrix.CreateScale(400,-300,1);
I was assuming this transformation order: (is it correct ?)
screenPoint = worldPoint*worldMatrix*projectionMatrix*viewMatrix
My motivation for this is that the projectionMatrix transforms the world box to normaliced device coordinates:(-10,-1)->(-1,1), (-2,-1)->(1,1), (-2,-7)->(1,-1), (-10,-7)->(-1,-1) and I then move this unity square to the fourth quadrant an scale it to screen size (800x600) and flip the y direction. But I am doing things wrong because I can't see nothing.
Resolved!
My problem was that I mistook the view transformation for a viewport transformation, i.e should map -1 .. 1 to screen.
But viewport transformations is never talked about in xna, they are implicit. This insight helped a lot. The solution turned out to be embarrassingly simple:
worldMatrix = Matrix.Identity;
viewMatrix = Matrix.Identity;
projectionMatrix = Matrix.CreateOrthographicOffCenter(-10, -2, -7, -1, -1.0f, 100.0f);