I got 2d coordinates of the rectangle(qr code) with the camera. Now, I need to convert these 2d coordinates into 3d coordinates with z index 0. As I understand I have to use camera pose estimation for this. But I could not find any special library that I can apply in rust. Is there any special library or sample code that already did this job?
Related
There are a number of calibration tutorials to calibrate camera images of chessboards in EMGU (OpenCV). They all end up calibrating and then undistorting an image for display. That's cool and all but I need to do machine vision where I am taking an image, identifying the location of a corner or blob or feature in the image and then translating the location of that feature in pixels into real world X, Y coordinates.
Pixel -> mm.
Is this possible with EMGU? If so, how? I'd hate to spend a bunch of time learning EMGU and then not be able to do this crucial function.
Yes, it's certainly possible as the "bread and butter" of OpenCV.
The calibration you are describing, in terms of removing distortions, is a prerequisite to this process. After which, the following applies:
The Intrinsic calibration, or "camera matrix" is the first of two required matrices. The second is the Extrinsic calibration of the camera which is essentially the 6 DoF transform that describes the physical location of the sensor center relative to a coordinate reference frame.
All of the Distortion Coefficients, Intrinsic, and Extrinsic Calibrations are available from a single function in Emgu.CV: CvInvoke.CalibrateCamera This process is best explained, I'm sure, by one of the many tutorials available that you have described.
After that it's as simple as CvInvoke.ProjectPoints to apply the transforms above and produce 3D coordinates from 2D pixel locations.
The key to doing this successfully this providing comprehensive IInputArray objectPoints and IInputArray imagePoints to CvInvoke.CalibrateCamera. Be sure to cause "excitation" by using many images, from many different perspectives.
I need to restore 3D shape from its 2D projection plot by recognizing it and finding 3d coordinates of vertices. You can see the sample images here. How can this be done most effectively? Is there any library that could do this? Language choice doesn't matter.
How important it is to do camera calibration for ArUco? What if I dont calibrate the camera? What if I use calibration data from other camera? Do you need to recalibrate if camera focuses change? What is the practical way of doing calibration for consumer application?
Before answering your questions let me introduce some generic concepts related with camera calibration. A camera is a sensor that captures the 3D world and project it in a 2D image. This is a transformation from 3D to 2D performed by the camera. Following OpenCV doc is a good reference to understand how this process works and the camera parameters involved in the same. You can find detailed AruCo documentation in the following document.
In general, the camera model used by the main libraries is the pinhole model. In the simplified form of this model (without considering radial distortions) the camera transformation is represented using the following equation (from OpenCV docs):
The following image (from OpenCV doc) illustrates the whole projection process:
In summary:
P_im = K・R・T ・P_world
Where:
P_im: 2D points porojected in the image
P_world: 3D point from the world
K is the camera intrinsics matrix (this depends on the camera lenses parameters. Every time you change the camera focus for exapmle the focal distances fx and fy values whitin this matrix change)
R and T are the extrensics of the camera. They represent the rotation and translation matrices for the camera respecively. These are basically the matrices that represent the camera position/orientation in the 3D world.
Now, let's go through your questions one by one:
How important it is to do camera calibration for ArUco?
Camera calibration is important in ArUco (or any other AR library) because you need to know how the camera maps the 3D to 2D world so you can project your augmented objects on the physical world.
What if I dont calibrate the camera?
Camera calibration is the process of obtaining camera parameters: intrinsic and extrinsic parameters. First one are in general fixed and depend on the camera physical parameters unless you change some parameter as the focus for example. In such case you have to re-calculate them. Otherwise, if you are working with camera that has a fixed focal distance then you just have to calculate them once.
Second ones depend on the camera location/orientation in the world. Each time you move the camera the RT matrices change and you have to recalculate them. Here when libraries such as ArUco come handy because using markers you can obtain these values automatically.
In few words, If you don't calculate the camera you won't be able to project objects on the physical world on the exact location (which is essential for AR).
What if I use calibration data from other camera?
It won't work, this is similar as using an uncalibrated camera.
Do you need to recalibrate if camera focuses change?
Yes, you have to recalculate the intrinsic parameters because the focal distance changes in this case.
What is the practical way of doing calibration for consumer application?
It depends on your application, but in general you have to provide some method for manual re-calibration. There're also method for automatic calibration using some 3D pattern.
I am doing a project on 3D rendering of a scene. I am using OpenCV. The steps I am doing are like this:
Taking two images of a scene.
Calculating object correspondence using SURF feature matching.
Calculating camera fundamental matrix.
Calculating the Disparity image.
Now I have two questions
After calculating fundamental matrix how can I calculate the Q matrix? (I can't calibrate the camera)
How can I render in 3D using opencv or any other library?
For the 3D part, you can render your scene with OpenGL or with PCL. You've two solutions:
For each pixel, you make a point with the right color extracted from the camera's image. This will give you a point cloud which can be processed with PCL (for 3D features extraction for example).
You apply a triangulation algorithm, but in order to apply this algorithm you must have the extrinsic matrices of your camera.
You can find more information about these techniques here:
Point Cloud technique
Triangulation algorithm
If you want to use OpenGL, you have to open a valid OpenGL context. I recommend you the SFML library or Qt. These libraries are very easy to use and have a good documentation. Both have tutorials about 3D rendering with OpenGL.
you can have Q matrix from stereo rectification via openCV method:
cv::stereoRectify
I think you want the Q matrix to reconstruct the 3D. However, you can reconstruct from intrinsic parameters via:
X = (u-cu)*base/d
Y = (v-cv)*base/d
Z = f*base/d
where (u,v) is a 2D point in the image coordinate system, (cu,cv) is the principal point of the camera, f is the focal length, base is the baseline, d is the disparity and (X,Y,Z) is a 3D point in the camera coordinate system.
For the visualization, it is possible to use PCL or VTK (the visualization of PCL is based on vtk, but for me more simple to implement).
If you just want to have a look to the output you can just use some software like Meshlab
Cheers
I am using OpenCV, a newbie to the entire thing.
I have a scenario, I am projecting on a wall, I am building a kind of a robot which has a camera. I wanted to know how can I process the image so that I could get the real-world values of the co-ordinates of the blobs tracked by my camera?
First of all, you need to calibrate the intrinsic of the camera. Use checkerboard-patterns printed on cardboard to do this, OpenCV has methods for this although there are finished tools for this as well.
To get an idea, I have written some python code to calibrate from a live video stream, move the cardboard along the camera in some different angles and distances. Take a look here: http://svn.ioctl.eu/pub/opencv/py-camera_intrinsic/
Then you need to calibrate the extrinsic of the camera, that is the position of the camera wrt. your world coordinates. You can place some markers on the wall, define the 3D-position of those markers and let OpenCV calibrate the extrinsic for this (cvFindExtrinsicCameraParams2).
In my sample code, I calculate the extrinsic wrt. the checkerboard so I can render a Teapot in the correct perspective of the camera. You have to adjust this to your needs.
I assume you project only onto a flat surface. You have to know the geometry to get the 3D coordinates of your detected blob. You can then find the blobs in your camera image and knowing intrinsic, extrinsic and the geometry, you can cast rays for each blob from the camera according to your intrinsic/extrinsic and calculate the intersection of each such ray with your known geometry. The intersection then is your 3D point in world space where the blob is projected to.