Compute homography for a virtual camera with opencv - opencv

I have an image of a planar surface, and I want to compute an image warping that gives me a synthetic view of the same planar surface seen from a virtual camera located at another point in the 3d space.
So, given an image I1 I want to compute an image I2 that represents the image I1 seen from a virtual camera.
In theory, there exists an homography that relates these two images.
How do I compute this homography given the camera pose of the virtual camera, as well as it's matrix of internal parameters?
I'm using opencv's warpPerspective() function to apply this homography and generate the image warped.
Thanks in advance.

Ok, found this post (Opencv virtually camera rotating/translating for bird's eye view), where I found some code doing what I needed.
However, I noticed that the rotation in Y had a sign error (-sin instead of sin) . Here's my solution adapted for python. I'm new to python, sorry if I'm doing something ugly.
import cv2
import numpy as np
rotXdeg = 90
rotYdeg = 90
rotZdeg = 90
f = 500
dist = 500
def onRotXChange(val):
global rotXdeg
rotXdeg = val
def onRotYChange(val):
global rotYdeg
rotYdeg = val
def onRotZChange(val):
global rotZdeg
rotZdeg = val
def onFchange(val):
global f
f=val
def onDistChange(val):
global dist
dist=val
if __name__ == '__main__':
#Read input image, and create output image
src = cv2.imread('/home/miquel/image.jpeg')
dst = np.ndarray(shape=src.shape,dtype=src.dtype)
#Create user interface with trackbars that will allow to modify the parameters of the transformation
wndname1 = "Source:"
wndname2 = "WarpPerspective: "
cv2.namedWindow(wndname1, 1)
cv2.namedWindow(wndname2, 1)
cv2.createTrackbar("Rotation X", wndname2, rotXdeg, 180, onRotXChange)
cv2.createTrackbar("Rotation Y", wndname2, rotYdeg, 180, onRotYChange)
cv2.createTrackbar("Rotation Z", wndname2, rotZdeg, 180, onRotZChange)
cv2.createTrackbar("f", wndname2, f, 2000, onFchange)
cv2.createTrackbar("Distance", wndname2, dist, 2000, onDistChange)
#Show original image
cv2.imshow(wndname1, src)
h , w = src.shape[:2]
while True:
rotX = (rotXdeg - 90)*np.pi/180
rotY = (rotYdeg - 90)*np.pi/180
rotZ = (rotZdeg - 90)*np.pi/180
#Projection 2D -> 3D matrix
A1= np.matrix([[1, 0, -w/2],
[0, 1, -h/2],
[0, 0, 0 ],
[0, 0, 1 ]])
# Rotation matrices around the X,Y,Z axis
RX = np.matrix([[1, 0, 0, 0],
[0,np.cos(rotX),-np.sin(rotX), 0],
[0,np.sin(rotX),np.cos(rotX) , 0],
[0, 0, 0, 1]])
RY = np.matrix([[ np.cos(rotY), 0, np.sin(rotY), 0],
[ 0, 1, 0, 0],
[ -np.sin(rotY), 0, np.cos(rotY), 0],
[ 0, 0, 0, 1]])
RZ = np.matrix([[ np.cos(rotZ), -np.sin(rotZ), 0, 0],
[ np.sin(rotZ), np.cos(rotZ), 0, 0],
[ 0, 0, 1, 0],
[ 0, 0, 0, 1]])
#Composed rotation matrix with (RX,RY,RZ)
R = RX * RY * RZ
#Translation matrix on the Z axis change dist will change the height
T = np.matrix([[1,0,0,0],
[0,1,0,0],
[0,0,1,dist],
[0,0,0,1]])
#Camera Intrisecs matrix 3D -> 2D
A2= np.matrix([[f, 0, w/2,0],
[0, f, h/2,0],
[0, 0, 1,0]])
# Final and overall transformation matrix
H = A2 * (T * (R * A1))
# Apply matrix transformation
cv2.warpPerspective(src, H, (w, h), dst, cv2.INTER_CUBIC)
#Show the image
cv2.imshow(wndname2, dst)
cv2.waitKey(1)

Related

how to project a point defined in real world coordinates to image plane and vice-versa?

I prepared a toy experiment to project a point defined in the world frame to the image plane. I'm trying to calculate the 3D point (inverse-projection) with the calculated pixel coordinates. I used the same coordinate frames as the figure below [https://www.researchgate.net/figure/World-and-camera-frame-definition_fig1_224318711]. (world x,y,z -> camera z,-x,-y). Then I will project the same point for different image frames. The problem here is that the 3D point defined as (8,-2.1) is calculated at (6.29, -1.60, 0.7). Since there is no loss of information, I think I need to find the same point. I believe there is a problem with depth. I couldn't find where I missed.
import numpy as np
#3d point at 8 -2 1 wrt world frame
P_world = np.array([8, -2, 1, 1]).reshape((4,1))
T_wc = np.array([
[ 0, -1, 0, 0],
[ 0, 0, -1, 0],
[ 1, 0, 0, 0],
[ 0, 0, 0, 1]])
pose0 = np.eye(4)
pose0[:3,-1] = [1, 0, .6]
pose0 = np.matmul(T_wc, pose0)
pose1 = np.eye(4)
pose1[:3,-1] = [3, 0, .6]
pose1 = np.matmul(T_wc, pose1)
depth0 = np.linalg.norm(P_world[:3].flatten() - np.array([1, 0, .6]).flatten())
depth1 = np.linalg.norm(P_world[:3].flatten() - np.array([3, 0, .6]).flatten())
K = np.array([
[173.0, 0 , 173.0],
[ 0 , 173.0, 130.0],
[ 0 , 0 , 1 ]])
uv1 = np.matmul(np.matmul(K, pose0[:3]), P_world)
uv1 = (uv1 / uv1[-1])[:2]
uv2 = np.matmul(np.matmul(K, pose1[:3]), P_world)
uv2 = (uv2 / uv2[-1])[:2]
img0 = np.zeros((260,346))
img0[int(uv1[0]), int(uv1[1])] = 1
img1 = np.zeros((260,346))
img1[int(uv2[0]), int(uv2[1])] = 1
#%% Op
pix_coord = np.array([int(uv1[0]), int(uv1[1]), 1])
pt_infilm = np.matmul(np.linalg.inv(K), pix_coord.reshape(3,1))
pt_incam = depth0*pt_infilm
pt_incam_hom = np.append(pt_incam, 1)
pt_inworld = np.matmul(np.linalg.inv(pose0), pt_incam_hom)

SVD not giving Rotation and Translation matrix from stereo essential matrix

I am trying to extract rotation and translation from the fundamental matrix. I used the intrinsic matrices to get the essential matrix, but the SVD is not giving expected results. So I composed the essential matrix and trying my SVD code to get the Rotation and translation matrices back and found that the SVD code is wrong.
I created the essential matrix using Rotation and translation matrices
R = [[ 0.99965657, 0.02563432, -0.00544263],
[-0.02596087, 0.99704732, -0.07226806],
[ 0.00357402, 0.07238453, 0.9973704 ]]
T = [-0.1679611706725666, 0.1475313058767286, -0.9746915198833979]
tx = np.array([[0, -T[2], T[1]], [T[2], 0, -T[0]], [-T[1], T[0], 0]])
E = R.dot(tx)
// E Output: [[-0.02418259, 0.97527093, 0.15178621],
[-0.96115177, -0.01316561, 0.16363519],
[-0.21769595, -0.16403593, 0.01268507]]
Now while trying to get it back using SVD.
U,S,V = np.linalg.svd(E)
diag_110 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 0]])
newE = U.dot(diag_110).dot(V.T)
U,S,V = np.linalg.svd(newE)
W = np.array([[0, -1, 0], [1, 0, 0], [0, 0, 1]])
Z = np.array([[0, 1, 0],[-1, 0, 0],[0, 0, 0]])
R1 = U.dot(W).dot(V.T)
R2 = U.dot(W.T).dot(V.T)
T = U.dot(Z).dot(U.T);
T = [T[1,0], -T[2, 0], T[2, 1]]
'''
Output
R1 : [[-0.99965657, -0.00593909, 0.02552386],
[ 0.02596087, -0.35727319, 0.93363906],
[-0.00357402, -0.93398105, -0.35730468]]
R2 : [[-0.90837444, -0.20840016, -0.3625262 ],
[ 0.26284261, 0.38971602, -0.8826297 ],
[-0.32522244, 0.89704559, 0.29923163]]
T : [-0.1679611706725666, 0.1475313058767286, -0.9746915198833979],
'''
What is wrong with the SVD code? I referred the code here and here
Your R1 output is a left-handed and axis-permuted version of your initial (ground-truth) rotation matrix: notice that the first column is opposite to the ground-truth, and the second and third are swapped, and that the determinant of R1 is ~= -1 (i.e. it's a left-handed frame).
The reason this happens is that the SVD decomposition returns unitary matrices U and V with no guaranteed parity. In addition, you multiplied by an axis-permutation matrix W. It is up to you to flip or permute the axes so that the rotation has the correct handedness. You do so by enforcing constraints from the images and the scene, and known order of the cameras (i.e. knowing which camera is the left one).

How to create a 2D perspective transform matrix from individual components?

I am trying to create a 2D perspective transform matrix from individual components like translation, rotation, scale, shear. But at the end the matrix is not producing a true perspective effect like the image below. I think I am missing some component in the code that I wrote to create the matrix. Could some one help me add the missing components and their formulation in the below function? I have used opencv library for my code
cv::Mat getPerspMatrix2D( double rz, double s, double tx, double ty ,double shx, double shy)
{
cv::Mat R = (cv::Mat_<double>(3,3) <<
cos(rz), -sin(rz), 0,
sin(rz), cos(rz), 0,
0, 0, 1);
cv::Mat S = (cv::Mat_<double>(3,3) <<
s, 0, 0,
0, s, 0,
0, 0, 1);
cv::Mat Sh = (cv::Mat_<double>(3,3) <<
1, shx, 0,
shy, 1, 0,
0, 0, 1);
cv::Mat T = (cv::Mat_<double>(3,3) <<
1, 0, tx,
0, 1, ty,
0, 0, 1);
return T * Sh * S * R;
}
Keywords are Homography and 8DOF. Taken from 1 and 2 there exists two coefficients for perspective transformation. But it needs a 2nd step to calculate it. I'm not familiar with OpenCV but I'm hoping to answer your question a bit late in a basically way ;-)
Step 1
You can imagine lx describes a vanishing point on the x axis. The image shows a31=lx=1. lx=100 is less transformation. For lx=0 the position is infinite far means no perspective transform = identity matrix.
[1 0 0]
PL = [0 1 0]
[lx ly 1]
lx/ly are perspective foreshortening parameters
Step 2
When you apply a right hand matrix multiplication P x [u; v; 1] you will recognize that the last value in the result is sometimes other than 1. For affine transformation it is always 1 for perspective projection not. In the 2nd step the result is scaled to make the last coefficient 1. This is a part of the effect.
Your Example Image
Image' = P4 x P3 x P2 x P1 x Image
I would translate the center of the blue rectangle to the origin tx=-w/2 and ty=-h/2 = P1.
Apply projective projection with ly = h (to make both sides at an angle)
Eventually translate back that all point are located in one quadrant
Eventually scale to desired size
Step 2 from the perspective projection can be done after 2.) or at the end.

OpenCV: Why projectPoints() is giving me weird results?

Why is my code snippet giving me weird results for projected points?
//Generate the one 3D Point which i want to project onto 2D plane
vector<Point3d> points_3d;
points_3d.push_back(Point3d(10, 10, 100));
Mat points3d = Mat(points_3d);
//Generate the identity matrix and zero vector for rotation matrix and translation vector
Mat rvec = (Mat_<double>(3, 3) << (1, 0, 0, 0, 1, 0, 0, 0, 1));
Mat tvec = (Mat_<double>(3, 1) << (0, 0, 0));
//Generate a camera intrinsic matrix
Mat K = (Mat_<double>(3,3)
<< (1000, 0, 50,
0, 1000, 50,
0, 0, 1));
//Project the 3D Point onto 2D plane
Mat points_2d;
projectPoints(points_3d, rvec, tvec, K, Mat(), points_2d);
//Output
cout << points_2d;
I get as projected 2D Point
points_2d = (-1.708699427820658e+024, -9.673395654445999e-026)
If i calculate it on a paper on my own, i'm expecting a point points_2d = (150, 150) with that formula:
Add cv::Rodrigues(InputArray src, OutputArray dst, OutputArray jacobian=noArray()). OpenCv uses rotation vector inside calculation instead of rotation matrix. Rodrigues transformation allows you to convert rotation vector to matrix and matrix to vector. Below i attached part of your code with one line added.
//Generate the identity matrix and zero vector for rotation matrix and translation vector
Mat rvec,rMat = (Mat_<double>(3, 3) << (1, 0, 0, 0, 1, 0, 0, 0, 1));
Rodrigues(rMat,rvec); //here
Mat tvec = (Mat_<double>(3, 1) << (0, 0, 0));
And it should work properly. It also will be better to define distortion coefficents as
Mat dist = Mat::zeros(8,1,CV_32f);
EDIT:
One more remark, you have little syntax error in matrix initialization:
cv::Mat rvec,rMat = (cv::Mat_<double>(3, 3) << /* ( */1, 0, 0, 0, 1, 0, 0, 0, 1); //you had error here
cv::Rodrigues(rMat, rvec);
cv::Mat tvec = (cv::Mat_<double>(3, 1) <</* ( */ 0, 0, 0); //and here
It works on my computer after changes.

Calculate new coordinates of keypoints after transformation

How to get the new coordinates of the points a and b in this exemple after a transformation M (40 degrees counter-clockwise rotation) ?
import cv2
cap = cv2.VideoCapture("http://i.imgur.com/7G91d2im.jpg")
a, b = (100, 100), (200, 200)
if cap.isOpened():
ret, im = cap.read()
rows, cols = im.shape[:2]
im_keypoints = im.copy()
for point in [a, b]:
cv2.circle(im_keypoints, point, 6, (0, 0, 255), -1)
cv2.imwrite("im_keypoints.jpg", im_keypoints)
M = cv2.getRotationMatrix2D((cols / 2, rows / 2), 40, 1)
im_rotated = cv2.warpAffine(im, M, (cols, rows))
cv2.imwrite("im_rotated.jpg", im_rotated)
M is a 2 by 3 rotation matrix, so all you need to do it apply M to your points.
im_rotated_keypoints = im_rotated.copy()
for point in [a, b]:
# Convert to homogenous coordinates in np array format first so that you can pre-multiply M
rotated_point = M.dot(np.array(point + (1,)))
cv.circle(im_rotated_keypoints, (int(rotated_point[0]), int(rotated_point[1])), 6, (0, 0, 255), -1)
And you should be able to see

Resources