OpenCV detecting TV screen using camera - opencv

Question

I am using an iPhone camera to detect a TV screen. My current approach is to compare subsequent frames pixel by pixel and keep track of cumulative differences. The result is binary a image as shown in image.
For me this looks like a rectangle but OpenCV does not think so. It's sides are not perfectly straight and sometimes there is even more color bleed to make detection difficult. Here is my OpenCV code trying to detect rectangle, since I am not very familiar with OpenCV it is copied from some example I found.
uint32_t *ptr = (uint32_t*)CVPixelBufferGetBaseAddress(buffer);
cv::Mat image((int)width, (int)height, CV_8UC4, ptr); // unsigned 8-bit values for 4 channels (ARGB)
cv::Mat image2 = [self matFromPixelBuffer:buffer];
std::vector<std::vector<cv::Point>>squares;
// blur will enhance edge detection
cv::Mat blurred(image2);
GaussianBlur(image2, blurred, cvSize(3,3), 0);//change from median blur to gaussian for more accuracy of square detection
cv::Mat gray0(blurred.size(), CV_8U), gray;
std::vector<std::vector<cv::Point> > contours;
// find squares in every color plane of the image
for (int c = 0; c < 3; c++) {
int ch[] = {c, 0};
mixChannels(&blurred, 1, &gray0, 1, ch, 1);
// try several threshold levels
const int threshold_level = 2;
for (int l = 0; l < threshold_level; l++) {
// Use Canny instead of zero threshold level!
// Canny helps to catch squares with gradient shading
if (l == 0) {
Canny(gray0, gray, 10, 20, 3); //
// Dilate helps to remove potential holes between edge segments
dilate(gray, gray, cv::Mat(), cv::Point(-1,-1));
} else {
gray = gray0 >= (l+1) * 255 / threshold_level;
}
// Find contours and store them in a list
findContours(gray, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
// Test contours
std::vector<cv::Point> approx;
int biggestSize = 0;
for (size_t i = 0; i < contours.size(); i++) {
// approximate contour with accuracy proportional
// to the contour perimeter
approxPolyDP(cv::Mat(contours[i]), approx, arcLength(cv::Mat(contours[i]), true)*0.02, true);
if (approx.size() != 4)
continue;
// Note: absolute value of an area is used because
// area may be positive or negative - in accordance with the
// contour orientation
int areaSize = fabs(contourArea(cv::Mat(approx)));
if (approx.size() == 4 && areaSize > biggestSize)
biggestSize = areaSize;
cv::RotatedRect boundingRect = cv::minAreaRect(approx);
float aspectRatio = boundingRect.size.width / boundingRect.size.height;
cv::Rect boundingRect2 = cv::boundingRect(approx);
float aspectRatio2 = (float)boundingRect2.width / (float)boundingRect2.height;
bool convex = isContourConvex(cv::Mat(approx));
if (approx.size() == 4 &&
fabs(contourArea(cv::Mat(approx))) > minArea &&
(aspectRatio >= minAspectRatio && aspectRatio <= maxAspectRatio) &&
isContourConvex(cv::Mat(approx))) {
double maxCosine = 0;
for (int j = 2; j < 5; j++) {
double cosine = fabs(angle(approx[j%4], approx[j-2], approx[j-1]));
maxCosine = MAXIMUM(maxCosine, cosine);
}
double area = fabs(contourArea(cv::Mat(approx)));
if (maxCosine < 0.3) {
squares.push_back(approx);
}
}
}
}
After Canny-step the image looks like this:
It seems fine to me but for some reason rectangle is not detected. Can anyone explain if there is something wrong with my parameters?
My second approach was to use OpenCV Hough line detection, basically using the same code as above, for Canny image I then call HoughLines function. It gives me quite a few lines as I had to lower threshold to detect vertical lines. The result looks like this:
The problem is that there are some many lines. How can I find out the lines that are touching the sides of blue rectangle as shown in first image?
Or is there a better approach to detect a screen?

Answer 1

First of all, find maximal area contour reference, then compure min area rectangle reference, divide contour area by rectangle area, if it close enough to 1 then your contour similar to rectangle. This will be your required contour and rectangle.