I want to detect pose of the Cube (see image). I have the coordinates of centers of all markers and I want to draw axis on the origin/ or outline of the whole cube using 2D detection points and 3D object points. I am using opencv and I tried to draw the axis, on the image but detected axis are not correct.
Aruco cube
I am using cv2.solvePnP, to get the rotation and translation vectors from detected 2D-3D point pairs.
Detected axis
for 3D object points Origin is at the one of the corners of the cube. The size of the cube is 20x20x20 cm.
Is it possible to do with opencv? Are there any other python libraries which could be useful to achieve the desired results?
TIA
Related
similar questions are solved many times. However, they generally maps depth coordinates to RGB coordinates, by following the next steps:
apply the inverse depth intrinsic matrix to the depth coordinates.
rotate and translate the 3d coordinates obtained using the rotation R and T matrixes that maps 3d depth coordinates to 3D RGB coordinates.
apply the RGB intrinsic matrix to obtain the image coordinates.
However, I want to do the reverse process. From a RGB coordinates obtain the depth coordinates. Then I can obtain an interpolated value from the depth map based on those coordinates.
The problem is that I don't know how can I define the z coordinate in the RGB image to make everything works.
The process should be:
obtain 3D RGB coordinates by applying the camera's inverse intrinsic matrix. How can I set the z coordinates? Should I define and estimated value? Set all the z coordinates to one?
rotate and translate the 3D RGB coordinates to the 3d coordinates.
apply the depth intrinsic matrix.
If this process cannot be done. How can I map RGB coordinates to depth coordinates instead of the other way around?
Thank you!
Considering the Sample Image I have Homography matrix 'h' which transforms the ground plane to the top view. The matrix is calculated using ArUco markers on the ground, so the matrix transforms the ground coordinates correctly. But now I want to transform the table's corner coordinates to the top view. The table plane is parallel to the ground in the 3D Coordinate system. We can assume an average distance between the ground plane and the table's plane to be 'd'. Is there any way to get the top view of the table or get a homography matrix for this? I have Camera intrinsic and extrinsic parameters, and the tomography matrix calculated for the ground plane.
I am thinking of first project table coordinates on the ground and then use the homography matrix we have, but I don't know if it is correct.
Sample Image
Look into the work that Antonio Criminisi did a looooong time ago on homologies between parallel planes. It's in his "Single View Metrology" paper. Complete treatment in his Ph.D. thesis.
I am trying to measure the precision of my marker tracking algorithm via post-processing a video.
My algorithm is: Find a printed planar marker in a Videostream and place a virtual marker at that position. I am working with AR.
Here are two frames of such a video:
Virtual Marker on top of detected marker
Virtual Marker with offset to actual marker
I want to calculate the Intersecion over Union / Jaccard Index of the actual marker and virtual marker. For the first picture it would give me ~98% and the second ~1/5th %. This will give me the quality for my algorithm, how precise and well it works.
I want to get the position and rotation of both markers in each frame with OpenCV and calculate the Jaccard Index. As you can see though, if I directly place a virtual marker on top of the paper marker, I will make it difficult for myself (with OpenCV) to detect them.
My idea is to not place a white marker on top of the actual marker, but place an easily detectable "thing" with a specific color or shape with an offset to the marker, let's say 10cm to the right maybe. Then I subtract the offset. So now, at the best case scenario, the position and rotation of the actual marker and the "thing" with the offset subtracted will be the same.
But what should I use as the easily detectable "thing"? I don't have enough experience with OpenCV to know what (colored?) shape I should use. The augmentation can go in front, behind, left, right... of the actual marker anytime during the video and it should do two things:
Not hinder the detection of the actual marker, like currently shown in the pictures
Be easily detectable itself
Help would be much appreciated!
Assuming you have enough white background around the visual marker:
You could use colored circles, for example in red, green, blue and black.
Use opencv blob detection [1] to detect all blobs and filter for circular ones:
Look-up average color values for detected blobs and filter for the colors of the circles.
Alternatively you could filter the whole image for each color and do blob detection on the filtered images. But this is slower.
Find the centroids (~ center point) of each blob using moments of the blob contours. [2] "Center of multiple blobs in an Image".
Now you have the four pixel positions of your circles. If you know the world coordinates of your light projected circles you can use solvePnP to get a pose from this.
Knowing the correct world coordinates is tricky in your case because you project the circle with light on a surface. This involves some 3D geometry. You need to know the transformation from camera coordinate system to pattern projector coordinate system and the projection parameters of your projector.
I guess you send the projected pattern as an image to the projector. I think you can then model the projector as a camera with a certain camera matrix (basically field of view & center point). Naturally you know the pixel coordinates of the projected circles. From this you can compute rays in 3D space (in projector coordinate system). As a starting point see [3]. Intersecting [4] them with the correct surface plane (in projector coordinate system) gives you the 3D coordinates of
the projected circle pattern in projector coordinate system. Transform these to camera coordinate system using your known transformation. Now use opencv solvePnP to determine pose of projected light marker.
How to get surface plane?
If your setup is static you could use visual marker detection of all recorded images and use mean oder median of marker pose as surface plane. Not sure what this implies for your evaluation though..
[1] https://www.learnopencv.com/blob-detection-using-opencv-python-c/
[2] https://www.learnopencv.com/find-center-of-blob-centroid-using-opencv-cpp-python/
[3] https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html
[4] https://www.cs.princeton.edu/courses/archive/fall00/cs426/lectures/raycast/sld017.htm
I am currently working on a camera 3D realsense camera that detection and calculate the box or boxes dimension.
I am new in computer vision. I first worked on i just work on detection objects detection with color or without color to get a basic understanding. Using C++ and openCV, I want to managed to get the corners (and their x y z pixel coordinates) of the square using smoothing (remove noise), edge detection (canny function), lines detection (Hough transform) and lines intersection (mathematical calculation) on an simplified picture (uniform background).
Now is my question: do you have any direction/recommendation/advice/literature about dimension calculation of box. https://www.youtube.com/watch?v=l-i2E7aZY6A
i am using c++ and opencv with Intel realsens 3D camera.
thanks in advance((-_-))
Once you have the colour image pixel coordinates of the box you can obtain the real-world coordinates (also known as vertices or camera coordinates in the documentation) using methods in the projection interface, then simple pythagoras to calculate the distance between the points in mm.
If you have no experience with RealSense I'd recommend reading the documentation and looking through the sample apps included with the SDK.
With PCL (Point Cloud Library) you can find planes (or spheres and other surfaces), then refine the result with 2D image processing (eg. edge detection).
http://pointclouds.org/
https://www.youtube.com/watch?v=VD044WAHEe4
I have a dataset of images with faces. I also have for each face within the dataset a set of 66 2D points that correspond to my face landmarks(nose, eyes, shape of my face, mouth).
So basically I have the shape of my face in terms of 2D points from my image.
Do you know any algorithm that I can use and that can rotate my shape so that the face shape is straight? Let's say that the pan angle is 30 degrees and I want it rotated to 30 degrees so that it is positioned at 0 degrees on the pan angle. I have illustrated bellow what I want to say.
Basically you can consider the above illustrated shapes outlines for my images, which are represented in 2D. I want to rotate my first shape points so that they can look like the second shape. A shape is made out of a set of 66 2D points which are basically pixel coordinates. All I want to do is to find the correspondence of each of those 66 points so that the new shape is rotated with a certain degree on the pan angle.
From your question, I can assume you either have the rotation parameters (e.g. degrees in x,y) or the point correspondences (since you have a database of matched points). Thus you either need to apply or estimate (and apply) a 2D similarity transformation for image alignment/registration. See also the response on this question: face alignment algorithm on images
From rotation angle and to new point locations: You can define a 2D rotation matrix R and transform your point coordinates with it.
From point correspondences between shape A and Shape B to rotation: Estimate a 2D similarity transform (image alignment) using 3 or more matching points.
From either rotation or point correspondences to warped image: From the similarity transform, map image values (accounting for interpolation or non-values) using the underlying coordinate transformation for the entire image grid.
(image courtesy of Denis Simakov, AAM Slides)
Most of these are already implemented in OpenCV and MATLAB. See also the background and relevant methods around Active Shape and Active Appearance Models (Tim Cootes page includes binaries and background material).