How to find the closest object to the camera? - opencv

Question

I am working on an algorithm which can find the point where the object is closest to the camera. I have found some help online by using meanshift. I have pasted the code to do that. How the code works is : it uses a track_window to track the closest object. But the problem is I don't know how to find the location of the closest object.
What I have tried:
I can make the track_window go through the full image and thus narrow down the area of interest but its too long and I still won't know the exact location.
Could someone help me find the closes object to the camera?
Here is my code:
/*
* back_project.cpp
*
* Created on: Mar 8, 2015
* Author: sushrut
*/
#include "backProject.h"
using namespace cv;
using namespace std;
/// Global Variables
Mat src; Mat hsv; Mat hue;
int bins = 25;
Rect track_window = cv::Rect(250, 250, 50, 50);
int x=1, y=1, wr=1, hr=1;
/// Function Headers
void Hist_and_Backproj(int, void* );
/** #function main */
int back_proj_main()
{
VideoCapture cap(0);
if(!cap.isOpened())
{
cout << "Cannot open the first web cam" << endl;
return -1;
}
while(true)
{
Mat src;
bool bSuccess = cap.read(src);
if(!bSuccess)
{
cout << "Cannot read a frame from video stream 1" << endl;
break;
}
/// Transform it to HSV
cvtColor( src, hsv, CV_BGR2HSV );
/// Use only the Hue value
hue.create( hsv.size(), hsv.depth() );
int ch[] = { 0, 0 };
mixChannels( &hsv, 1, &hue, 1, ch, 1 );
/// Create Trackbar to enter the number of bins
char* window_image = "Source image";
namedWindow( window_image, CV_WINDOW_AUTOSIZE );
createTrackbar("* Hue bins: ", window_image, &bins, 180, Hist_and_Backproj );
Hist_and_Backproj(0, 0);
// create a rectangle on the image
rectangle(src, Point(x,y), Point(x+wr, y+hr), Scalar(0, 250, 0));
/// Show the image
imshow( window_image, src );
if (waitKey(30) == 27) //wait for 'esc' key press for 30ms. If 'esc' key is pressed, break loop
{
cout << "esc key is pressed by user" << endl;
break;
}
}
// TODO release the capture
return 0;
}
/**
* #function Hist_and_Backproj
* #brief Callback to Trackbar
*/
void Hist_and_Backproj(int, void* )
{
MatND hist;
int histSize = MAX( bins, 2 );
float hue_range[] = { 0, 180 };
const float* ranges = { hue_range };
/// Get the Histogram and normalize it
calcHist( &hue, 1, 0, Mat(), hist, 1, &histSize, &ranges, true, false );
normalize( hist, hist, 0, 255, NORM_MINMAX, -1, Mat() );
/// Get Backprojection
MatND backproj;
calcBackProject( &hue, 1, 0, hist, backproj, &ranges, 1, true );
/// Draw the backproj
imshow( "BackProj", backproj );
// code to create a meanshift
meanShift(backproj,track_window, TermCriteria(TermCriteria::COUNT|TermCriteria::EPS, 10, 1));
// update the value of recctange
x = track_window.x;
y = track_window.y;
wr = track_window.width;
hr = track_window.height;
cout << "values: " << x << ", " << y << ", " << wr << ", " << hr << endl;
/// Draw the histogram
int w = 400; int h = 400;
int bin_w = cvRound( (double) w / histSize );
Mat histImg = Mat::zeros( w, h, CV_8UC3 );
for( int i = 0; i < bins; i ++ )
{ rectangle( histImg, Point( i*bin_w, h ), Point( (i+1)*bin_w, h - cvRound( hist.at<float>(i)*h/255.0 ) ), Scalar( 0, 0, 255 ), -1 ); }
imshow( "Histogram", histImg );
}
Refrences:
http://www.opencv.org.cn/opencvdoc/2.3.1/html/doc/tutorials/imgproc/histograms/back_projection/back_projection.html

Answer 1

Generally, it is impossible to detect which object is closest to the camera if you have only one frame and one camera - object relative position.
You should process video sequence and track feature points to determine positions of objects in the coordinate frame of the camera.
You should study some materials on multiple view geometry. For example, lectures of professor Daniel Cremers on YouTube. A book by Hartley and Zisserman is also very good.
Update after reading the comment
It is unclear what points you mean. Are they pixel coordinates of the object in the image or points in 3D Euclidean space?
If first, then you can proceed in the following. Backproject returns you an image, where each pixel contains a probability of belonging to the object. You could threshold this image to throw out very low probabilities, filter the result from noise, refine with morphological operators (erosion, closing) thus getting a blob of pixels, most probably belonging to you object. Then you can find its bounding rectangle. If you do not want to use backproject, because it doesn't always find the object, then you can use 2D feature detectors and matchers from OpenCV. They can estimate locations of some "extreme" points, belonging to the object.
In the second case the task looks like a task of SLAM (Simultaneous Localization and Mapping). It is more complex subject. I again reference you to multiview geometry and SLAM courses. This task also heavily uses 2D feature points.