I want to develop an AR application use unity and OpenCV. I use solvePNP method in openCV to get camera rotation matrix and translation as follows:
r11 r12 r13 tx
r21 r22 r23 ty
r31 r32 r33 tz
In unity, projectionMatrix and WorldToCameraMatrix are 4x4 Matrix,which correspond to camera instrinsic matrix and camera pose
In Order to align two coordinate system. I set
WorldToCameraMatrix=
r11 r12 r13 tx
-r21 -r22 -r23 -ty
-r31 -r32 -r33 -tz
0 0 0 1
and I set
projectMatrix =
2*f/w 0 0 0
0 2*f/h 0 0
0 0 -(far+near)/(far-near)) -2*(far*near)/(far-near)
0 0 -1 0
After these , the rotation and translation is correct.But the cube warps heavily.
As this image：
cube warps
Who can help me to find the reason?Thanks in advance.
Related
I'm trying to make a freecam sort of tool for a rather obscure game. I'm trying to figure out how to move the XY position of the freecam based on the car's rotation value, but I'm struggling to do so. I've tried using Calculating X Y movement based on rotation angle? and modifying it a bit, but it doesn't work as intended. The game's rotation uses a float that ranges from -1 to 1, -1 being 0 degrees and 1 being 360 degrees.
Putting rot at -1 corresponds to X+
Putting rot at 0 corresponds to Z+
Putting rot at 1 corresponds to X-
Here's my cheat engine code:
speed = 10000
local yaw = math.rad((180*(getAddressList().getMemoryRecordByDescription('rot').Value))+180)
local px = getAddressList().getMemoryRecordByDescription('playerx').Value
local py = getAddressList().getMemoryRecordByDescription('playery').Value
local pz = getAddressList().getMemoryRecordByDescription('playerz').Value
local siny = math.sin(yaw) -- Sine of Horizontal (Yaw)
local cosy = math.cos(yaw) -- Cosine of Horizontal (Yaw)
getAddressList().getMemoryRecordByDescription('playerx').Value = ((getAddressList().getMemoryRecordByDescription('playerx').Value)+(cosy*speed))
getAddressList().getMemoryRecordByDescription('playerz').Value = ((getAddressList().getMemoryRecordByDescription('playerz').Value)+(siny*speed))
print(yaw)
in opencv, i know i can use Perspective Transformation to do in this way:
pts1 = np.float32([[56,65],[368,52],[28,387],[389,390]])
pts2 = np.float32([[0,0],[300,0],[0,300],[300,300]])
M = cv2.getPerspectiveTransform(pts1,pts2)
dst = cv2.warpPerspective(img,M,(300,300))
https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html
then i can rotate the image by 90 degree by rotate function.
cv::rotate(image, image, ROTATE_90_CLOCKWISE);
But i think i should be able to do the two steps in one step.
How should I create the matrix to multiple with PerspectiveTransform M,
i thought about this:
0 -1 0
1 0 0
0 0 1
but this is wrong.
note: I will do it in Java.
I am trying to estimate a gues for the intrinsic matrix, K, of a DJI Phantom 4 drone. I know that the form of this matrix is:
but i cant seem to get the units right. Looking up the specs at https://www.dji.com/phantom-4/info#specs I find that the focal length is 8.88 (dosnt say units...) and the image dimensions are 4000x3000. WHat would K look like with these?
*PS, I am scaling down the images so they are smaller. Will this effect the K matrix I should use for openCV?
The page the OP linked to lists a FOV of 94 degrees. With an image width of 4000 pixels this corresponds to a focal length of
f = (4000 / 2) pixels / tan(94 / 2 degrees) = 1865 pixels
Absent any other calibration data, one should therefore use an estimated camera matrix of the form:
K = [ [1865, 0 , 2000],
[0 , 1865, 1500],
[0 , 0 , 1 ] ]
OP, you may have confused the specs of the P4 and the P4Pro, which have different sensors and lenses. The P4Pro, not the P4, has a focal length of 8.8mm. The P4 has a focal length of 3.61mm.
If you are indeed using images from a P4, Francesco's answer is correct.
However, if you are actually using images from a P4Pro, you need to use these values:
f = (4864 / 2) pixels / tan(84 / 2 degrees) = 2701 pixels
K = [ [2701, 0 , 2432],
[0 , 2701, 1824],
[0 , 0 , 1 ] ]
For future reference for anyone that may find this answer, here are the relevant specs for the P4 and P4Pro sensors/lenses:
Phantom 4:
Sensor size: 1/2.3" (6.17mm x 4.55mm)
Focal length (actual): 3.61mm
Focal length (35mm equivalent): 20mm
FOV: 94°
Image size: 4000×3000 pixels
Video frame size
UHD: 4096×2160 pixels
4K: 3840×2160 pixels
2.7K: 2704×1520 pixels
FHD: 1920×1080 pixels
HD: 1280×720 pixels
Phantom 4 Pro:
Sensor size: 1" (12.8mm x 9.6mm)
Focal length (actual): 8.88mm
Focal length (35mm equivalent): 24mm
FOV: 84°
Image size
3:2 aspect ratio: 5472×3648 pixels
4:3 aspect ratio: 4864×3648 pixels
16:9 aspect ratio: 5472×3078 pixels
Video frame size
C4K: 4096×2160 pixels
4K: 3840×2160 pixels
2.7K: 2720×1530 pixels
FHD: 1920×1080 pixels
HD: 1280×720 pixels
I think it is much better to work from the focal length in mm
https://www.dxomark.com/Cameras/DJI/Phantom4-Pro---Specifications
For P4 Pro:
13.2 x 8.8 so pixel size is = 0.00241 or 2.41 um focal length is 8.8mm
so focal length in pixel = 8.8 / 0.00241 = 3684.6 pixels
Incidentally in the image metadata, there is a field:
CalibratedFocalLength 3666.666504 (use exiftool to find it) so I think K should be
K = [ [3666.6, 0 , 2432],
[0 , 3666.6, 1824],
[0 , 0 , 1 ] ]
I have the camera intrinsics matrix and the distortion coefficients from OpenCV camera calibration.
Am an wondering where the skew factor of the calibration is contained in this Matrices and how I can get it as a float or float[].
Skew is by default set to 0 at least since OpenCV 2.0.
If you look at the camera matrix it looks like A = [fx skew u0; 0 fy v0; 0 0 1]
but in OpenCv it is A = [fx 0 u0; 0 fy v0; 0 0 1].
Suppose I have the coordinates X, Y, Z and orientation Rx, Ry, Rz of an object with respect to a camera.
In addition, I have the coordinates U, V, W and orientation Ru, Rv, Rw of this camera in the world.
How do I transform the position (location and rotation) of the object to its position in the world?
It sounds like a change of basis to me, but I haven't found a clear source yet.
I have found this document which is quite clear on the topic.
http://www.cse.psu.edu/~rcollins/CSE486/lecture12.pdf
It treats, among others, the reverse operation, i.e., going from world to camera 3D coordinates.
Pc = R ( Pw - C )
Where, Pc is a point in the camera world, Pw is a point in the normal world, R is a rotation matrix and C is the camera translation.
Unfortunately it is rather cumbersome to add latex formulae, so I will give some matlab code instead.
function lecture12_collins()
% for plotting simplicity I choose my points on plane z=0 in this example
% Point in the world
Pw = [2 2.5 0 1]';
% rotation
th = pi/3;
% translation
c = [1 2.5 0]';
% obtain world to camera coordinate matrix
T = GetT(th, c);
% calculate the camera coordinate
Pc = T*Pw
% get the camera to world coordinate
T_ = GetT_(th, c)
% Alternatively you could use the inverse matrix
% T_ = inv(R*C)
% calculate the worldcoordinate
Pw_ = T_*Pc
assert (all(eq(Pw_ ,Pw)))
function T = GetT(th, c)
% I have assumed rotation around the z axis only here.
R = [cos(th) -sin(th) 0 0
sin(th) cos(th) 0 0
0 0 1 0
0 0 0 1];
C = [1 0 0 -c(1)
0 1 0 -c(2)
0 0 1 -c(3)
0 0 0 1];
T = R*C;
function T_ = GetT_(th, c)
% negate the angle
R_ = [cos(-th) -sin(-th) 0 0
sin(-th) cos(-th) 0 0
0 0 1 0
0 0 0 1];
% negate the translation
C_ = [1 0 0 c(1)
0 1 0 c(2)
0 0 1 c(3)
0 0 0 1];
T_ = C_*R_
So far this is about the location only. The rotation I've solved by using extra knowledge I had of the rotation. I know that my camera is perpendicular to the object and that its rotation is only around the z axis. I can just add the rotation of the camera and the object.
In fact you have two basis : one relative to the camera, the other is absolute (the world). So you basically want to transform your relative data into absolute data.
Location
This is the easiest one. You have to translate the (X,Y,Z) position by the vector t(U,V,W). So all your positions in absolute are (Ax, Ay, Az) = (X,Y,Z)+t = (X+U,Y+V,Z+W).
Orientation
This is a bit more difficult. You have to find the rotation matrix that rotate your camera from (I assume) (0,0,1) to (Ru,Rv,Rw). You should look at Basic Rotation Matrices in order to decompose the 2 rotations that take (0,0,1) to (Ru,Rv,Rw) (one according to X axis, one according to Z axis for example). I advise you to draw the absolute basis and the vector (Ru,Rv,Rw) on a sheet of paper, it is the simplest way to get the right result.
So you have 2 basic rotations matrices r1 and r2. The resultant rotation matrix r = r1*r2 (or r2*r1, it doesn't matter). So the absolute orientation of your object is (ARx, ARy, ARz)
= r*(Rx,Ry,Rz).
Hope this helps !