Related
I am running solvePnPRansac on an image dataset, with 2d feature points and triangulated 3d landmark points. It runs great, and the results in rotation, and in the forward and side axes, look great. The Y axis though, is completely wrong.
I am testing the output against the ground truth from the data set, and it goes up where it should go down, and drifts off the ground truth very quickly. The other axes stay locked on for much much longer.
this strikes me as strange, how can it be correct for the other axes, and wrong for one? Surely that is not possible, I would have thought that either every axis was bad, or every axis was good.
What could i possibly be doing wrong to make this happen? And how can i debug this weirdness? My PnP code is very standard:
cv::Mat inliers;
cv::Mat rvec = cv::Mat::zeros(3, 1, CV_64FC1);
int iterationsCount = 500; // number of Ransac iterations.
float reprojectionError = 2.0; //2.0 // maximum allowed distance to consider it an inlier.
float confidence = 0.95; // RANSAC successful confidence.
bool useExtrinsicGuess = false;
int flags = cv::SOLVEPNP_ITERATIVE;
int num_inliers_;
//points3D_t0
cv::solvePnPRansac(points3D_t0, points_left_t1, intrinsic_matrix, distCoeffs, rvec, translation_stereo,
useExtrinsicGuess, iterationsCount, reprojectionError, confidence,
inliers, flags);
I encountered similar problem for images taking by a drone – sometimes the Y value (camera line of sight axis – the height axis in my case) was BELOW the ground. If you think about it – for a plane view (or close to a plane) - there are two possible ‘y’ solutions : before the plane and away to the plane (below and under the ground in my case). So both are a legal solutions.
I'm trying out D3D11 and struggling to render a model correctly.
Here's my problem; while my world and view transformations seem right,
my perspective transformation seems to be wrong.
When I first rendered a model, something felt wrong, so I tried rotating the model to see what it was.
Then I noticed that, parts of the model closer to the camera appears smaller, and further parts appear larger.
If it's relevant, I'm using assimp to load my model, and here's how I do it.
mScene = aiImportFile(filename.c_str(), aiProcessPreset_TargetRealtime_MaxQuality | aiProcess_GenSmoothNormals | aiProcess_ConvertToLeftHanded | aiProcess_TransformUVCoords);
And here's how I build my projection matrix.
mProjection = XMMatrixPerspectiveFovLH(XMConvertToRadians(45.0f), 800.0f / 600.0f, -1.0f, 1.0f);
I fiddled with nearZ and farZ arguments of XMMatrixPerspectiveFovLH.
I tried increasing farZ gradually every frame, and then realized that as the value increases, the far clipping plane comes closer and closer to the camera, which is exactly the opposite of what I thought would happen.
In vertex shader, here's what I'm doing with vertex positions. It's pretty basic.
Out.Position = mul(mul(mul(position, World), CameraView), CameraProjection);
The model renders correctly in terms of position, scaling, rotation, and view-position.
So I'm assuming that world and view transforms are fine, and the problem is about the projection matrix.
To summarize, I'm thinking that Z values of projected vertices are, somehow, "flipped".
I google-searched many many times to no avail.
If someone could point out what I could be doing wrong, it would be very much appreciated.
If you need to see some of my code to help, please tell me.
Your near and far plane distances should be positive.
Use something like:
mProjection = XMMatrixPerspectiveFovLH(XMConvertToRadians(45.0f), 800.0f / 600.0f, 0.1f, 10.0f);
I'll make a note to consider adding assert( NearZ > 0.f); assert( NearZ < FarZ ); to those DirectXMath functions and make sure that's explicit in the docs. distance means positive number here.
PS: You should take a look at DirectX Tool Kit
I'm trying to detect objects that are similar to circles using OpenCV's HoughCircles. The problem is: HoughCircles fails to detect such objects in some cases.
Does anyone know any alternative way to detect objects similar to circles like these ones?
Update
Update
Hello Folks I'm adding a gif of the result of my detection method.
It's easier use a gif to explain the problem. The undesired effect that I want to remove is the circle size variation. Even for a static shape like the one on the right, the result on the left is imprecise. Does anyone know a solution for that?
Update
All that I need from this object is its diameter. I've done it using findContours. Now I can't use findContours once it is too slow when using openCV and OpenMP. Does anyone know a fast alternatives to findContours?
Update
The code that I'm using to detect these shapes.
for (int j=0; j<=NUM_THREADS-1;j++)
{
capture >> frame[j];
}
#pragma omp parallel shared(frame,processOutput,circles,diameterArray,diameter)
{
int n=omp_get_thread_num();
cvtColor( frame[n], processOutput[n], CV_BGR2GRAY);
GaussianBlur(processOutput[n], processOutput[n], Size(9, 9), 2, 2);
threshold(processOutput[n], processOutput[n], 21, 250, CV_THRESH_BINARY);
dilate(processOutput[n], processOutput[n], Mat(), Point(-1, -1), 2, 1, 1);
erode(processOutput[n], processOutput[n], Mat(), Point(-1, -1), 2, 1, 1);
Canny(processOutput[n], processOutput[n], 20, 20*2, 3 );
HoughCircles( processOutput[n],circles[n], CV_HOUGH_GRADIENT, 1, frame[n].rows/8, 100,21, 50, 100);
}
#pragma omp parallel private(m, n) shared(circles)
{
#pragma omp for
for (n=0; n<=NUM_THREADS-1;n++)
{
for( m = 0; m < circles[n].size(); m++ )
{
Point center(cvRound(circles[n][m][0]), cvRound(circles[n][m][2]));
int radius = cvRound(circles[n][m][3]);
diameter = 2*radius;
diameterArray[n] = diameter;
circle( frame[0], center, 3, Scalar(0,255,0), -1, 8, 0 );
circle( frame[0], center, radius, Scalar(0,0,255), 3, 8, 0 );
}
}
}
Edited based on new description and additional performance and accuracy requirements.
This is getting beyond the scope of an "OpenCV sample project", and getting into the realm of actual application development. Both performance and accuracy become requirements.
This requires a combination of techniques. So, don't just pick one approach. You will have to try all combinations of approaches, as well as fine-tune the parameters to find an acceptable combination.
#1. overall approach for continuous video frame recognition tasks
Use a slow but accurate method to acquire an initial detection result.
Once a positive detection is found on one frame, the next frame should switch to a fast local search algorithm using the position detected on the most recent frame.
As a reminder, don't forget to update the "most recent position" for use by the next frame.
#2. suggestion for initial object acquisition.
Stay with your current approach, and incorporate the suggestions.
You can still fine-tune the balance between speed and precision, because a correct but imprecise result (off by tens of pixels) will be updated and refined when the next frame is processed with the local search approach.
Try my suggestion of increasing the dp parameter.
A large value of dp reduces the resolution at which Hough Gradient Transform is performed. This reduces the precision of the center coordinates, but will improve the chance of detecting a dented circle because the dent will become less significant when the transform is performed at a lower resolution.
An added benefit is that reduced resolution should run faster.
#3. suggestion for fast local search around a previously detected position
Because of the limited search space and amount of data needed, it is possible to make local search both fast and precise.
For tracking the movement of the boundary of iris through video frames, I suggest using a family of algorithms called the Snakes model.
The focus is on tracking the movement of edges through profiles. There are many algorithms that can implement the Snakes model. Unfortunately, most implementations are tailored to very complex shape recognition, which would be an overkill and too slow for your project.
Basic idea: (assuming that the previous result is a curve)
Choose some sampling points on the curve.
Scan the edge profile (perpendicular to the curve) at each the sampling point, on the new frame, using the position of the old frame. Look for the sharpest change.
Remember the new edge position for this sampling point.
After all of the sampling points have been updated, create a new curve by joining all of the updated sampling point positions.
There are many varieties, and different levels of sophistication of implementations which you can find on the Internet. Unfortunately, it was reported that the one packaged with OpenCV might not work very well. You may have to try different open-source implementation, and ultimately you may have to implement one that is simple but well-tuned to your project's needs.
#4. Seek advice for your speed optimization attempts.
Use a software performance profiler.
Add some timing and logging code around each call to OpenCV function to print out the time spent on each step. You will be surprised. The reason is that some OpenCV functions are more heavily vectorized and parallelized than others, perhaps as a result of the labor of love.
Unfortunately, for the slowest step - initial object acquisition, there is not much you can parallelize (by multithread).
This is perhaps already obvious to you since you did not put #pragma omp for around the first block of code. (It would not help anyway.)
Vectorization (SIMD) would only benefit pixel-level processing. If OpenCV implements it, great; if not, there is not much you can do.
My guess is that cvtColor, GaussianBlur, threshold, dilate, erode could have been vectorized, but the others might not be.
Give try on below,
Find contour in source.
Find minimum enclosing circle for the contour.
Now draw contour to new Mat with CV_FILLED.
Similarly draw enclosing circle to new Mat with filled option.
Perform x-or operation between the above two and count non-zero.
You can decide the contour is close to circle or not by comparing the non-zero pixel between contour and enclosimg circle with a threshold. You can decide the threshold by calculating the area of encosing circle, and taking it percent.
The idea is simple the area between contour and its enclosing circle decreases as the contour closes to circle
I've got problem with precise detection of markers using OpenCV.
I've recorded video presenting that issue: http://youtu.be/IeSSW4MdyfU
As you see I'm markers that I'm detecting are slightly moved at some camera angles. I've read on the web that this may be camera calibration problems, so I'll tell you guys how I'm calibrating camera, and maybe you'd be able to tell me what am I doing wrong?
At the beginnig I'm collecting data from various images, and storing calibration corners in _imagePoints vector like this
std::vector<cv::Point2f> corners;
_imageSize = cvSize(image->size().width, image->size().height);
bool found = cv::findChessboardCorners(*image, _patternSize, corners);
if (found) {
cv::Mat *gray_image = new cv::Mat(image->size().height, image->size().width, CV_8UC1);
cv::cvtColor(*image, *gray_image, CV_RGB2GRAY);
cv::cornerSubPix(*gray_image, corners, cvSize(11, 11), cvSize(-1, -1), cvTermCriteria(CV_TERMCRIT_EPS+ CV_TERMCRIT_ITER, 30, 0.1));
cv::drawChessboardCorners(*image, _patternSize, corners, found);
}
_imagePoints->push_back(_corners);
Than, after collecting enough data I'm calculating camera matrix and coefficients with this code:
std::vector< std::vector<cv::Point3f> > *objectPoints = new std::vector< std::vector< cv::Point3f> >();
for (unsigned long i = 0; i < _imagePoints->size(); i++) {
std::vector<cv::Point2f> currentImagePoints = _imagePoints->at(i);
std::vector<cv::Point3f> currentObjectPoints;
for (int j = 0; j < currentImagePoints.size(); j++) {
cv::Point3f newPoint = cv::Point3f(j % _patternSize.width, j / _patternSize.width, 0);
currentObjectPoints.push_back(newPoint);
}
objectPoints->push_back(currentObjectPoints);
}
std::vector<cv::Mat> rvecs, tvecs;
static CGSize size = CGSizeMake(_imageSize.width, _imageSize.height);
cv::Mat cameraMatrix = [_userDefaultsManager cameraMatrixwithCurrentResolution:size]; // previously detected matrix
cv::Mat coeffs = _userDefaultsManager.distCoeffs; // previously detected coeffs
cv::calibrateCamera(*objectPoints, *_imagePoints, _imageSize, cameraMatrix, coeffs, rvecs, tvecs);
Results are like you've seen in the video.
What am I doing wrong? is that an issue in the code? How much images should I use to perform calibration (right now I'm trying to obtain 20-30 images before end of calibration).
Should I use images that containg wrongly detected chessboard corners, like this:
or should I use only properly detected chessboards like these:
I've been experimenting with circles grid instead of of chessboards, but results were much worse that now.
In case of questions how I'm detecting marker: I'm using solvepnp function:
solvePnP(modelPoints, imagePoints, [_arEngine currentCameraMatrix], _userDefaultsManager.distCoeffs, rvec, tvec);
with modelPoints specified like this:
markerPoints3D.push_back(cv::Point3d(-kMarkerRealSize / 2.0f, -kMarkerRealSize / 2.0f, 0));
markerPoints3D.push_back(cv::Point3d(kMarkerRealSize / 2.0f, -kMarkerRealSize / 2.0f, 0));
markerPoints3D.push_back(cv::Point3d(kMarkerRealSize / 2.0f, kMarkerRealSize / 2.0f, 0));
markerPoints3D.push_back(cv::Point3d(-kMarkerRealSize / 2.0f, kMarkerRealSize / 2.0f, 0));
and imagePoints are coordinates of marker corners in processing image (I'm using custom algorithm to do that)
In order to properly debug your problem I would need all the code :-)
I assume you are following the approach suggested in the tutorials (calibration and pose) cited by #kobejohn in his comment and so that your code follows these steps:
collect various images of chessboard target
find chessboard corners in images of point 1)
calibrate the camera (with cv::calibrateCamera) and so obtain as a result the intrinsic camera parameters (let's call them intrinsic) and the lens distortion parameters (let's call them distortion)
collect an image of your own custom target (the target is seen at 0:57 in your video) and it is shown in the following figure and find some relevant points in it (let's call the point you found in image image_custom_target_vertices and world_custom_target_vertices the corresponding 3D points).
estimate the rotation matrix (let's call it R) and the translation vector (let's call it t) of the camera from the image of your own custom target you get in point 4), with a call to cv::solvePnP like this one cv::solvePnP(world_custom_target_vertices,image_custom_target_vertices,intrinsic,distortion,R,t)
giving the 8 corners cube in 3D (let's call them world_cube_vertices) you get the 8 2D image points (let's call them image_cube_vertices) by means of a call to cv2::projectPoints like this one cv::projectPoints(world_cube_vertices,R,t,intrinsic,distortion,image_cube_vertices)
draw the cube with your own draw function.
Now, the final result of the draw procedure depends on all the previous computed data and we have to find where the problem lies:
Calibration: as you observed in your answer, in 3) you should discard the images where the corners are not properly detected. You need a threshold for the reprojection error in order to discard "bad" chessboard target images. Quoting from the calibration tutorial:
Re-projection Error
Re-projection error gives a good estimation of just how exact is the
found parameters. This should be as close to zero as possible. Given
the intrinsic, distortion, rotation and translation matrices, we first
transform the object point to image point using cv2.projectPoints().
Then we calculate the absolute norm between what we got with our
transformation and the corner finding algorithm. To find the average
error we calculate the arithmetical mean of the errors calculate for
all the calibration images.
Usually you will find a suitable threshold with some experiments. With this extra step you will get better values for intrinsic and distortion.
Finding you own custom target: it does not seem to me that you explain how you find your own custom target in the step I labeled as point 4). Do you get the expected image_custom_target_vertices? Do you discard images where that results are "bad"?
Pose of the camera: I think that in 5) you use intrinsic found in 3), are you sure nothing is changed in the camera in the meanwhile? Referring to the Callari's Second Rule of Camera Calibration:
Second Rule of Camera Calibration: "Thou shalt not touch the lens
after calibration". In particular, you may not refocus nor change the
f-stop, because both focusing and iris affect the nonlinear lens
distortion and (albeit less so, depending on the lens) the field of
view. Of course, you are completely free to change the exposure time,
as it does not affect the lens geometry at all.
And then there may be some problems in the draw function.
So, I've experimented a lot with my code, and I still haven't fixed the main issue (shifted objects), but I've managed to answer some of calibration questions I've asked.
First of all - in order to obtain good calibration results you have to use images with properly detected grid elements/circles positions!. Using all captured images in calibration process (even those that aren't properly detected) will result bad calibration.
I've experimented with various calibration patterns:
Asymmetric circles pattern (CALIB_CB_ASYMMETRIC_GRID), give much worse results than any other pattern. By worse results I mean that it produces a lot of wrongly detected corners like these:
I've experimented with CALIB_CB_CLUSTERING and it haven't helped much - in some cases (different light environment) it got better, but not much.
Symmetric circles pattern (CALIB_CB_SYMMETRIC_GRID) - better results than asymmetric grid, but still I've got much worse results than standard grid (chessboard). It often produces errors like these:
Chessboard (found using findChessboardCorners function) - this method is producing best possible results - it doesn't produce misaligned corners very often, and almost every calibration is producing similar results to best-possible results from symmetric circles grid
For every calibration I've been using 20-30 images that were coming from different angles. I've tried even with 100+ images but it haven't produced noticeable change in calibration results than smaller amount of images. It's worth noticing that larger number of test images is increasing time needed to compute camera parameters in non-linear way (100 test images in 480x360 resolution are computing 25 minutes in iPad4, compared with 4 minutes with ~50 images)
I've also experimented with solvePNP parameters - but is also haven't gave me any acceptable results: I've tried all 3 detection methods (ITERATIVE, EPNP and P3P), but I haven't seen aby noticeable change.
Also I've tried with useExtrinsicGuess set to true, and I've used rvec and tvec from previous detection, but this one resulted with complete disapperance of detected cube.
I've ran out of ideas - what else could be affecting these shifting problems?
For those still interested:
this is an old question, but I think your problem is not the bad calibration.
I developed an AR app for iOS, using OpenCV and SceneKit, and I have had your same issue.
I think your problem is the wrong render position of the cube:
OpenCV's solvePnP returns the X, Y, Z coordinates of the marker center, but you wanna render the cube over the marker, at a specific distance along the Z axis of the marker, exactly at one half of the cube side size. So you need to improve the Z coordinate of the marker translation vector of this distance.
In fact, when you see your cube from the top, the cube is render properly.
I have done an image in order to explain the problem, but my reputation prevent to post it.
I am trying to do normal mapping on flat surface but I can't get any noticeable result :(
My shader
http://pastebin.com/raEvBY92
For my eye, shader looks fine, but it doesn't render desired result( https://dl.dropbox.com/u/47585151/sss/final.png).
All values are passed.Normals,tengents and binormals are computed correctly when I create the grid,I have checked that!
Here are screens of ambient,diffuse,specular and bump map.
https://dl.dropbox.com/u/47585151/sss/ambient.png
https://dl.dropbox.com/u/47585151/sss/bumpMap.png
https://dl.dropbox.com/u/47585151/sss/diffuse.png
https://dl.dropbox.com/u/47585151/sss/specular.png
They seems to be legit...
The bump map,which is the result of (bump=normalize(mul(bump, input.WorldToTangentSpace)) definitely looks correct,but doesn't have any impact on end result.
Maybe I don't understand the different spaces idea or I changed the order of matrix multiplication.By world matrix I understand the position and orientation of the grid,which never changes and it is identity matrix.Only view matrix changes and represents camera position and orientation in own space.
Where is my mistake?
First of all, if you're having a problem, it's a good ideo to comment everything out, which doesn't belong to this. The whole lightcomputation with ambient, specular or even the diffusetexture isn't interesting at this moment. With
output.color=float4(diffuse ,1);
You can focus on your problem and see clearly what change, if you change something in you code.
If your quad lies in the xy-plane with z=0, you should change your lightvector, he wouldn't work. Generally I use for testing purpose a diagonal vector (like normalize(1,-1,1)) to prevent a parallel direction to my object.
When I look over your code it seems to me, that you didn't get the idea of the different spaces, as how you thought ;) The basic idea of normalmapping is to give additional information about the surface with additional normals. They are saved in a normalmap, so encoded to rgb, where b is usually the up-vector. Now you must fit them into your 3D-world, because they aren't in the world space, but in the tangent space (tangent space = surface space of the triangle). Because this transformation is more complex, the normal computation goes the other way round. You transform with the normal,binormal and tangent as a matrix your lightvector and viewvector from world space into tangent space (you are mapping the axis of world space xyz to tnb - tangent,normal,binormal, the order can be wrong I usually swap them until it works ;) ). With your line
bump = normalize(mul(bump, input.WorldToTangentSpace));
you try to transform you normal in tangent space to tangent space. Change this, so you transform the view and the lightvector in the vertexshader into tangent space and pass the transformed vectors to the pixelshader. There you can do the lightcomputation in tangent space. Maybe read an additional tutorial to normalmapping, then you will get this working! :)
PS: If youre finished with the basic lighting, your specular computation seems to have some errors, too.
float3 reflect = normalize(2*diffuse*bump-LightDirection);
This line seems to should compute the halfway-vector, but therefore you need the viewvector and shouldn't use a lightingstrength like diffuse. But a tutorial can explain this in more detail than me now.