A few years ago, Apple started warning anyone using GLKit in their app that OpenGL was going away:
warning: OpenGLES is deprecated. Consider migrating to Metal instead
warning: GLKit is deprecated. Consider migrating to MetalKit instead
My app uses a complex OpenGL class, and I don't know either OpenGL or Metal. Apple has a few WWDC sessions on this, but they are targeted at OpenGL experts. Since Apple is going to remove OpenGL someday, I want to start this now before I only have a few months to do it. What should I do?
tldr;
Once I started seeing the build error messages in iOS12:
warning: OpenGLES is deprecated. Consider migrating to Metal instead
warning: GLKit is deprecated. Consider migrating to MetalKit instead
I knew I had to do something. Who knows exactly when Apple will remove OpenGL and GLKit? Trust me, you don't want to wait until you have just a few months to convert to Metal, as the process is by no means straight forward.
What follows is the process I used to convert
an Objective-C/OpenGL view into Metal. It was a long arduous process and several times
I put my head on my desk and cried in frustration.
The fundamental steps I took were ones I would suggest others adopt too:
Remove all business logic and anything not directly related to OpenGL from the view, and restructure the primary app as necessary.
Create a test harness app that you will use for the conversion, and absolutely put it under version control.
Add the OpenGL view to the test harness.
Once the ViewController can drive the view, and you can see it, you are ready to start the transition.
In my case, I had three hurtles to jump: convert the view to Swift, recreate the functionality in Metal, then replace all GLK vector and matrix values and operations to simd.
My suggestion for proceeding:
Convert any ObjectiveC to Swift (I used Swiftify, free for limited translation, however I had a subscription)
Add a MTKView to the test harness, and put code switches in the ViewController so as to alternately look at either view (comparing both was a big help to me).
Since I didn't know either OpenGL or Metal, I spent a lot of time downloading open source Metal projects and tutorials.
Create the Metal boilerplate (based on examples/tutorials) along with a shader.
Put a pad on your desk so when you bang your head in frustration trying to get anything to show in the Metal view you don't seriously hurt yourself.
Once you are over the hill, convert the GLK values/operations to simd, making use of the translation functions shown later.
I cannot stress this enough—commit every time you change a few operations and test them! You will surely break things and that way you can reference earlier working code.
The test harness will prove useful, as you will likely find that timing changes result in undesired behavior. In my case I created two harnesses, a second that had more of the app code in it so I could better debug the actual usage.
Project
I forked an open source project Panorama. The master branch contains the Metal/simd code, and the Swift-OpenGL branch contains the original ObjectiveC code along with the Swift conversion. This lets a reader compare the two side-by-side. However, you don't need to referernce that to gleen much of how to convert OpenGL code in ObjectiveC to Swift, or to convert GLKit vectors and matrices to simd, as follows.
ObjectiveC to Swift
The OpenGL code makes much use of pointers, and those are a bit more burdensome in Swift. For instance:
GLfloat *m_TexCoordsData; // treated as an array of pairs of floats
glTexCoordPointer(2, GL_FLOAT, 0, m_TexCoordsData);
became
struct Pointer2 {
private var array: [SIMD2<Float>]
init(size: Int) {
let n: SIMD2<Float> = [Float.nan, Float.nan]
array = Array<SIMD2<Float>>(repeating: n, count: size)
}
subscript(index: Int) -> SIMD2<Float>{
get { return array[index] }
set(newValue) { array[index] = newValue }
}
mutating func usingRawPointer(block: WithRawPtr) {
array.withUnsafeBytes { (bufPtr) -> Void in
block(bufPtr.baseAddress!)
}
}
}
private var tPtr: Pointer2 // m_TexCoordsData
tPtr.usingRawPointer(block: { (ptr) in
glTexCoordPointer(2, UInt32(GL_FLOAT), 0, ptr)
})
This all went away in the final Metal code.
Doing the conversion turned up latent bugs too!
In addition, I had to cast (convert) many of the values to stricter Swift types:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
became
glTexParameteri(UInt32(GL_TEXTURE_2D), UInt32(GL_TEXTURE_WRAP_S), GLint(GL_REPEAT))
and this process was just tedious. However, the Swift code was thinner and easier to read IMHO.
A couple of functions translated easily:
GLKQuaternion GLKQuaternionFromTwoVectors(GLKVector3 u, GLKVector3 v) {
GLKVector3 w = GLKVector3CrossProduct(u, v);
GLKQuaternion q = GLKQuaternionMake(w.x, w.y, w.z, GLKVector3DotProduct(u, v));
q.w += GLKQuaternionLength(q);
return GLKQuaternionNormalize(q);
}
became
func GLKQuaternionFromTwoVectors(_ u: GLKVector3, _ v: GLKVector3) -> GLKQuaternion {
let w = GLKVector3CrossProduct(u, v)
var q = GLKQuaternionMake(w.x, w.y, w.z, GLKVector3DotProduct(u, v))
q.w += GLKQuaternionLength(q)
return GLKQuaternionNormalize(q)
}
Later you will see that the translation to simd was not so easy.
OpenGL to Metal
Unfortunately, there is no magic wand here. Apple has some WWDC sessions on this, but they didn't really enlighten me. Metal uses two types of kernels: compute and shader, with compute the easier. However in my case I had to use a shader, which was more difficult for me to grasp.
Metal Resources
A good place to start if you know nothing of Metal is this Metal Tutorial on the Ray Wenderlich site. A second article there is even more helpful: Moving From OpenGL to Metal on the Ray Wenderlich site. Both have copious references to more Metal material.
Two other excellent articles I found helpful: Donald Pinckney's Blog (Older). Another helpful author: Alex Barbulescu
The guy who literally wrote the book on Metal is Warren Moore. His book and articles are invaluable!
Things to keep in mind
OpenGL uses a clip space of -1 to 1 (z values). You need to account for this in your shader. Warren Moore personally suggested I insure my shader was not returning negative z values by using this code:
v.z = v.z * 0.5 + v.w * 0.5;
This obviates the need to completely redo your OpenGL code that might have used negative z values.
The backgroundColor of a MTLView is not set by using that property, but setting the clearColor.
The communication from App space to shader space is done using structures, which must be separately defined in each respectively. For instance, in my app this struct:
private struct Uniforms {
let projectionMatrix: simd_float4x4
let attitudeMatrix: simd_float4x4
}
is defined in the shader as:
struct Uniforms {
float4x4 projectionMatrix;
float4x4 attitudeMatrix;
};
These structs are application defined.
Textures
If you are using images to create textures, it's a bit different in Metal. This
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:YES], GLKTextureLoaderOriginBottomLeft, nil];
GLKTextureInfo *info=[GLKTextureLoader textureWithContentsOfFile:path options:options error:&error];
glBindTexture(GL_TEXTURE_2D, info.name);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, IMAGE_SCALING);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, IMAGE_SCALING);
became
let loader: MTKTextureLoader = MTKTextureLoader(device: mtlDevice)
do {
let texture = try loader.newTexture(cgImage: cgImage, options: [
MTKTextureLoader.Option.origin: MTKTextureLoader.Origin.bottomLeft,
MTKTextureLoader.Option.SRGB: false // yes and image washed out
])
return texture
} catch {
Note that you need to set the origin to bottomLeft.
Final Comments
Unless you plan on really learning Metal deeply, you will be doing a lot of experimentation and asking questions. Having a test harness app will prove invaluable as you spend hours trying to get your code to do what you want.
Translate GLK to simd
Any Apple code is surely full of GLKVectors, GLKMatrices, and related functions. Unfortunately, there are no tools to convert them—you must do it by hand, line by line, and sometimes there is not simd equivalent. Sometime I used Xcode's search and replace, but not often.
GLK -> SIMD
First, to get the simd macros, add this to your source files: import simd
GLfloat -> Float
GLint -> Int
GLKMatrix4 -> simd_float4 (typealias for SIMD4)
GLKMatrix4Identity -> matrix_identity_float4x4 (not easy to find)
GLKMatrix4Invert -> simd_inverse(simd_float4x4)
GLKMatrix4Make -> simd_float4x4(simd_float4, simd_float4, simd_float4, simd_float4)
GLKMatrix4MakeFrustum -> no replacement, function provided below
GLKMatrix4MakeLookAt -> no replacement, function provided below
GLKMatrix4MakeWithQuaternion -> simd_matrix4x4(simd_quatf)
GLKMatrix4Multiply -> simd_float4x4 * simd_float4x4
GLKMatrix4MultiplyVector3 -> no replacement, function provided below
GLKMatrix4MultiplyVector4 ->simd_float4x4 * simd_float4
GLKQuaternion -> simd_quatf
GLKQuaternionLength -> simd_quatf.length
GLKQuaternionMake -> simd_quaternion(_ x: Float, _y: Float, _ z: Float, _ w: Float)
GLKQuaternionNormalize -> simd_quatf.normalized
GLKTextureInfo -> MTLTexture
GLKVector3 -> simd_float3
GLKVector3CrossProduct -> simd_cross(simd_float3, simd_float3)
GLKVector3DotProduct -> simd_dot(simd_float3, simd_float3)
GLKVector3Make -> simd_make_float3(_ x: Float, _y: Float, _ z: Float)
GLKVector3Normalize -> simd_normalize(simd_float3)
GLKVector4 -> simd_float4
GLKVector4Make -> simd_make_float4(_ x: Float, _y: Float, _ z: Float, _ w: Float)
I should note that Dash was a tremendous help in digging through the simd functions.
The two functions referenced above:
func simd_make_look_at_float4x4(
eyeX: Float,
eyeY: Float,
eyeZ: Float,
centerX: Float,
centerY: Float,
centerZ: Float,
upX: Float,
upY: Float,
upZ: Float
) -> simd_float4x4 {
// https://stackoverflow.com/questions/9053377/ios-questions-about-camera-information-within-glkmatrix4makelookat-result
let ev = simd_float3(eyeX, eyeY, eyeZ)
let cv = simd_float3(centerX, centerY, centerZ)
let uv = simd_float3(upX, upY, upZ)
let subbed = ev - cv
let n = simd_normalize(subbed)
let cross_p = simd_cross(uv, n)
let u = simd_normalize(cross_p)
let v = simd_cross(n, u)
let c0: simd_float4 = [u[0], v[0], n[0], 0]
let c1: simd_float4 = [u[1], v[1], n[1], 0]
let c2: simd_float4 = [u[2], v[2], n[2], 0]
let v0 = simd_dot(-1*u, ev)
let v1 = simd_dot(-1*v, ev)
let v2 = simd_dot(-1*n, ev)
let c3: simd_float4 = [v0, v1, v2, 1]
let m: simd_float4x4 = simd_float4x4(columns: (c0, c1, c2, c3))
return m
}
func simd_make_frustum_float4x4(frustum: Float, aspectRatio: Float) -> simd_float4x4 {
// https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glFrustum.xml
let left = -frustum
let right = frustum
let bottom = -frustum/aspectRatio
let top = frustum/aspectRatio
let near = PanoramaView.Z_NEAR
let far = PanoramaView.Z_FAR
let m00 = (2.0 * near) / (right - left)
let m11 = (2.0 * near) / (top - bottom)
let m20 = (right + left) / (right - left)
let m21 = (top + bottom) / (top - bottom)
let m22 = -1 * (far + near) / (far - near)
let m23 = Float(-1)
let m32 = -1 * (2 * far * near) / (far - near)
let c0: simd_float4 = [m00, 0, 0, 0]
let c1: simd_float4 = [0, m11, 0, 0]
let c2: simd_float4 = [m20, m21, m22, m23]
let c3: simd_float4 = [0, 0, m32, 0]
let m = simd_float4x4(columns: (c0, c1, c2, c3))
return m
}
// Translated from the original Panorama code
func simd_make_quaternion_from_two_vectors(_ u: simd_float3, _ v: simd_float3) -> simd_quatf {
let w: simd_float3 = simd_cross(u, v)
var q: simd_quatf = simd_quaternion(w.x, w.y, w.z, simd_dot(u, v))
q.real += q.length
return q.normalized
}
Translate back and forth between GLK and simd
These functions are found in the Panorama repository mentioned earlier, in a file GLK-Metal-Tools.swift. If as recommended you translate back and forth after your controller is solely simd, you can put these in your view to as you slowly remove the GLK code.
func glkV3_to_simd(_ v3: GLKVector3) -> simd_float3 {
let v: simd_float3 = simd_make_float3(v3.x, v3.y, v3.z)
return v
}
func simd3_to_glk(_ v3: simd_float3) -> GLKVector3 {
let v = GLKVector3Make(v3[0], v3[1], v3[2])
return v
}
func glkV4_to_simd(_ v3: GLKVector4) -> simd_float4 {
let v: simd_float4 = simd_make_float4(v3.x, v3.y, v3.z, v3.w)
return v
}
func simd4x4_to_glk(_ m: simd_float4x4) -> GLKMatrix4 {
var array: [GLKVector4] = []
for i in 0..<4 {
let fv: simd_float4 = m[i]
let v: GLKVector4 = GLKVector4Make(fv[0], fv[1], fv[2], fv[3]);
array.append(v)
}
let mg: GLKMatrix4 = GLKMatrix4MakeWithColumns(array[0], array[1], array[2], array[3]);
return mg;
}
func glkm4_to_simd(_ m: GLKMatrix4) -> simd_float4x4 {
var array: [simd_float4] = []
for i in 0..<4 {
let fv: GLKVector4 = GLKMatrix4GetColumn(m, Int32(i))
let v: simd_float4 = simd_make_float4(fv[0], fv[1], fv[2], fv[3]);
array.append(v)
}
let ms: simd_float4x4 = simd_matrix(array[0], array[1], array[2], array[3]);
return ms;
}
I used these print routines to check various values during development, you might find them of use too:
func print4x4SIMD(
msg: String,
m: simd_float4x4
) {
var s = ""
s += "---COL: \(msg)\n"
let (c0, c1, c2, c3) = m.columns
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 0, c0[0], c0[1], c0[2], c0[3])
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 1, c1[0], c1[1], c1[2], c1[3])
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 2, c2[0], c2[1], c2[2], c2[3])
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 3, c3[0], c3[1], c3[2], c3[3])
print("\n\(s)\n")
}
func print4x4GLK(
msg: String,
m: GLKMatrix4
) {
var s = ""
s += "---COL: \(msg)\n"
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 0, m.m00, m.m01, m.m02, m.m03)
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 1, m.m10, m.m11, m.m12, m.m13)
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 2, m.m20, m.m21, m.m22, m.m23)
s += String(format: "[%.2d] %10.4lf %10.4lf %10.4lf %10.4lf\n", 3, m.m30, m.m31, m.m32, m.m33)
print("\n\(s)\n")
}
simd_float4x4 rotation
While I didn't use this yet, I may need it someday (it's untested):
func matrix_from_rotation(radians: Float, v _v: simd_float3) -> simd_float4x4 {
// https://www.haroldserrano.com/blog/rotating-a-2d-object-using-metal
let v: simd_float3 = simd_normalize(_v)
let cos: Float = cos(radians)
let cosp: Float = 1 - cos
let sin: Float = sin(radians)
let col0 = simd_float4(
cos + cosp * v.x * v.x,
cosp * v.x * v.y + v.z * sin,
cosp * v.x * v.z - v.y * sin,
0
)
let col1 = simd_float4(
cosp * v.x * v.y - v.z * sin,
cos + cosp * v.y * v.y,
cosp * v.y * v.z + v.x * sin,
0.0
)
let col2 = simd_float4(
cosp * v.x * v.z + v.y * sin,
cosp * v.y * v.z - v.x * sin,
cos + cosp * v.z * v.z,
0.0
)
let col3 = simd_float4(0, 0, 0, 1)
let m: simd_float4x4 = simd_float4x4(columns: (col0, col1, col2, col3))
return m
}
Conclusion
This project took me about half a year working one weekend day per week, but it spanned a period of 18 months. The reason: I spent so many days making no progress, getting bizarre corrupted output, or no output, that when I finally got the primary view to show in Metal as it did in, I put the project away. I was just too burned out to go on.
That said, the end of in iOS is going to end, and as the years roll by that end is getting nearer.
I was originally going to stop when I got Metal working with the GLK vectors and matrices, but was urged to convert to simd now by Warren Moore.
It was a moment of pure ecstasy when I finally built my company's app and there wasn't a single compiler warning related to GLKit!
I want to implement stereo vision in a robot. I have calculated disparity map and point clouds. now I want to detect Dynamic Obstacles in scene.
Can anyone help me please?
Best Regards
Here is how I do that for 2d navigation.
First prepare two 2d elevation maps as 2d arrays. Set elements of one of the arrays to min height of points projected to the same cell of the 2d map, and set elements of the other array to max heights like this:
static const float c_neg_inf = -9999;
static const float c_inf = 9999;
int map_pixels_in_m_ = 40; //: for map cell size 2.5 x 2.5 cm
int map_width = 16 * map_pixels_in_m_;
int map_height = 16 * map_pixels_in_m_;
cv::Mat top_view_min_elevation(cv::Size(map_width, map_height), CV_32FC1, cv::Scalar(c_inf));
cv::Mat top_view_max_elevation(cv::Size(map_width, map_height), CV_32FC1, cv::Scalar(c_neg_inf));
//: prepare elevation maps:
for (int i = 0, v = 0; v < height; ++v) {
for (int u = 0; u < width; ++u, ++i) {
if (!pcl::isFinite(point_cloud_->points[i]))
continue;
pcl::Vector3fMap point_in_laser_frame = point_cloud_->points[i].getVector3fMap();
float z = point_in_laser_frame(2);
int map_x = map_width / 2 - point_in_laser_frame(1) * map_pixels_in_m_;
int map_y = map_height - point_in_laser_frame(0) * map_pixels_in_m_;
if (map_x >= 0 && map_x < map_width && map_y >= 0 && map_y < map_width) {
//: update elevation maps:
top_view_min_elevation.at<float>(map_x, map_y) = std::min(top_view_min_elevation.at<float>(map_x, map_y), z);
top_view_max_elevation.at<float>(map_x, map_y) = std::max(top_view_max_elevation.at<float>(map_x, map_y), z);
}
}
}
Then
//: merge values in neighboring pixels of the elevation maps:
top_view_min_elevation = cv::min(top_view_min_elevation, CvUtils::hscroll(top_view_min_elevation, -1, c_inf));
top_view_max_elevation = cv::max(top_view_max_elevation, CvUtils::hscroll(top_view_max_elevation, -1, c_neg_inf));
top_view_min_elevation = cv::min(top_view_min_elevation, CvUtils::hscroll(top_view_min_elevation, 1, c_inf));
top_view_max_elevation = cv::max(top_view_max_elevation, CvUtils::hscroll(top_view_max_elevation, 1, c_neg_inf));
top_view_min_elevation = cv::min(top_view_min_elevation, CvUtils::vscroll(top_view_min_elevation, -1, c_inf));
top_view_max_elevation = cv::max(top_view_max_elevation, CvUtils::vscroll(top_view_max_elevation, -1, c_neg_inf));
top_view_min_elevation = cv::min(top_view_min_elevation, CvUtils::vscroll(top_view_min_elevation, 1, c_inf));
top_view_max_elevation = cv::max(top_view_max_elevation, CvUtils::vscroll(top_view_max_elevation, 1, c_neg_inf));
Here CvUtils::hscroll and CvUtils::vscroll are functions that 'scroll' the content of a 2d array filling the elements on the edge that got no value in the scroll with the value of the third parameter.
Now you can make a difference between the arrays (taking care about elements with c_inf and c_neg_inf values) like this:
//: produce the top_view_elevation_diff_:
cv::Mat top_view_elevation_diff = top_view_max_elevation - top_view_min_elevation;
cv::threshold(top_view_elevation_diff, top_view_elevation_diff, c_inf, 0, cv::THRESH_TOZERO_INV);
Now all non-zero elements of top_view_elevation_diff are your potential obstacles. You can enumerate them and report 2d coordinates of those of them that are grater then some value as your 2d obstacles.
If you can wait till the middle of September, I'll put into a public repository full code of a ROS node that takes a depth image and depth camera info and generates a faked LaserScan message with measurements set to distance to found obstacles.
I need to calculate a warp matrix in OpenCV, representing the rotation around a given axis.
Around Z axis -> straightforward: I use the standard rotation matrix
[cos(a) -sin(a) 0]
[sin(a) cos(a) 0]
[ 0 0 1]
This is not so obvious for other rotations, so I tried building a homography, as explained on Wikipedia:
H = R - t n^T / d
I tried with a simple rotation around the X axis, and assuming the distance between the camera and the image is twice the image height.
R is the standard rotation matrix
[1 0 0]
[0 cos(a) -sin(a)]
[0 sin(a) cos(a)]
n is [0 0 1]because the camera is looking directly at the image from (0, 0, z_cam)
t is the translation, that should be [0 -2h*(sin(a)) -2h*(1-cos(a))]
d is the distance, and it's 2h per definition.
So, the final matrix is:
[1 0 0]
[0 cos(a) 0]
[0 sin(a) 1]
which looks quite good, when a = 0 it's an identity, and when a = pi it's mirrored around the x axis.
And yet, using this matrix for a perspective warp doesn't yield the expected result, the image is just "too warped" for small values of a, and disappears very quickly.
So, what am I doing wrong?
(note: I've read many questions and answers about this subject, but all are going in the opposite direction: I don't want to decompose a homography matrix, but rather to build one, given a 3d transformation, and a "fixed" camera or a fixed image and a moving camera).
Thank you.
Finally found a way, thanks to this post: https://plus.google.com/103190342755104432973/posts/NoXQtYQThgQ
I let OpenCV calculate the matrix for me, but I'm doing the perspective transform myself (found it easier to implement than putting everything into a cv::Mat)
float rotx, roty, rotz; // set these first
int f = 2; // this is also configurable, f=2 should be about 50mm focal length
int h = img.rows;
int w = img.cols;
float cx = cosf(rotx), sx = sinf(rotx);
float cy = cosf(roty), sy = sinf(roty);
float cz = cosf(rotz), sz = sinf(rotz);
float roto[3][2] = { // last column not needed, our vector has z=0
{ cz * cy, cz * sy * sx - sz * cx },
{ sz * cy, sz * sy * sx + cz * cx },
{ -sy, cy * sx }
};
float pt[4][2] = {{ -w / 2, -h / 2 }, { w / 2, -h / 2 }, { w / 2, h / 2 }, { -w / 2, h / 2 }};
float ptt[4][2];
for (int i = 0; i < 4; i++) {
float pz = pt[i][0] * roto[2][0] + pt[i][1] * roto[2][1];
ptt[i][0] = w / 2 + (pt[i][0] * roto[0][0] + pt[i][1] * roto[0][1]) * f * h / (f * h + pz);
ptt[i][1] = h / 2 + (pt[i][0] * roto[1][0] + pt[i][1] * roto[1][1]) * f * h / (f * h + pz);
}
cv::Mat in_pt = (cv::Mat_<float>(4, 2) << 0, 0, w, 0, w, h, 0, h);
cv::Mat out_pt = (cv::Mat_<float>(4, 2) << ptt[0][0], ptt[0][1],
ptt[1][0], ptt[1][1], ptt[2][0], ptt[2][1], ptt[3][0], ptt[3][1]);
cv::Mat transform = cv::getPerspectiveTransform(in_pt, out_pt);
cv::Mat img_in = img.clone();
cv::warpPerspective(img_in, img, transform, img_in.size());
I'm trying to find the orientation of a binary image (where orientation is defined to be the axis of least moment of inertia, i.e. least second moment of area). I'm using Dr. Horn's book (MIT) on Robot Vision which can be found here as reference.
Using OpenCV, here is my function, where a, b, and c are the second moments of area as found on page 15 of the pdf above (page 60 of the text):
Point3d findCenterAndOrientation(const Mat& src)
{
Moments m = cv::moments(src, true);
double cen_x = m.m10/m.m00; //Centers are right
double cen_y = m.m01/m.m00;
double a = m.m20-m.m00*cen_x*cen_x;
double b = 2*m.m11-m.m00*(cen_x*cen_x+cen_y*cen_y);
double c = m.m02-m.m00*cen_y*cen_y;
double theta = a==c?0:atan2(b, a-c)/2.0;
return Point3d(cen_x, cen_y, theta);
}
OpenCV calculates the second moments around the origin (0,0) so I have to use the Parallel Axis Theorem to move the axis to the center of the shape, mr^2.
The center looks right when I call
Point3d p = findCenterAndOrientation(src);
rectangle(src, Point(p.x-1,p.y-1), Point(p.x+1, p.y+1), Scalar(0.25), 1);
But when I try to draw the axis with lowest moment of inertia, using this function, it looks completely wrong:
line(src, (Point(p.x,p.y)-Point(100*cos(p.z), 100*sin(p.z))), (Point(p.x, p.y)+Point(100*cos(p.z), 100*sin(p.z))), Scalar(0.5), 1);
Here are some examples of input and output:
(I'd expect it to be vertical)
(I'd expect it to be horizontal)
I worked with the orientation sometimes back and coded the following. It returns me the exact orientation of the object. largest_contour is the shape that is detected.
CvMoments moments1,cenmoments1;
double M00, M01, M10;
cvMoments(largest_contour,&moments1);
M00 = cvGetSpatialMoment(&moments1,0,0);
M10 = cvGetSpatialMoment(&moments1,1,0);
M01 = cvGetSpatialMoment(&moments1,0,1);
posX_Yellow = (int)(M10/M00);
posY_Yellow = (int)(M01/M00);
double theta = 0.5 * atan(
(2 * cvGetCentralMoment(&moments1, 1, 1)) /
(cvGetCentralMoment(&moments1, 2, 0) - cvGetCentralMoment(&moments1, 0, 2)));
theta = (theta / PI) * 180;
// fit an ellipse (and draw it)
if (largest_contour->total >= 6) // can only do an ellipse fit
// if we have > 6 points
{
CvBox2D box = cvFitEllipse2(largest_contour);
if ((box.size.width < imgYellowThresh->width) && (box.size.height < imgYellowThresh->height))
{
cvEllipseBox(imgYellowThresh, box, CV_RGB(255, 255 ,255), 3, 8, 0 );
}
}
my current issue is the following: I try to create a SpriteBatch with integrated Primtive-Rendering (Sprite and PrimtiveBatch in one).
Currently Visual Studio 11's Debugger shows that there is a line rendered. The IA-Stage is correct, so the VertexShader-Stage is. But i don´t get any output.
I run over it with the debugger and found out, that the shader has abnormal huge values.
I´m passing two vertices: 20,20 (Position) and 80, 80 (Position) both colored black.
The ConstantBuffer has three matrices: World (for manual object transformation), View and Projection.
In code, they´re all correct:
M11 = 0.00215749722, and so on.
In the shader, it looks like this:
Position = x = 200000.000000000, y = 200000.000000000, z = 0.000000000, w = 0.000000000
(#1 vertex's position)
Projection[0] = x = 22.000000000, y = 0.000000000, z = 0.000000000, w = 0.000000000
Projection[1] x = 0.000000000, y = -45.000000000, z = 0.000000000, w = 0.000000000
Projection[2] x = 0.000000000, y = 0.000000000, z = 10000.000000000, w = 0.00000 0000
Projection[3] x = 0.000000000, y = 0.000000000, z = 0.000000000, w = 10000.000000000
And finally the output result (position) is 0 for XYZW.
Before it was 413,-918 or something like that. I don´t know why, but now the result is always 0...
This is very strange, and i´m totaly stuck :-/
Thanks
R