openCV problem with detecting contours of shapes fully - opencv

Question

I am doing this university project where i try to detect UI elements on screenshots of Android applications using openCV. I am not expecting a 100 percent accuracy for this detection of UI elements.
This is my code below. I convert the image to gray scale, apply Gaussian blur and then use adaptive threshold to convert the image to binary. After which i use the find contours method.
ap = argparse.ArgumentParser()
ap.add_argument("-i","--image", help = "path to an image", required =
True)
args = vars(ap.parse_args())
image = cv2.imread(args["image"])
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imshow("gray",gray)
cv2.waitKey(0)
blurred = cv2.GaussianBlur(gray, (5,5), 0)
thresh = cv2.adaptiveThreshold(blurred, 255,
cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 11, 4)
cv2.imshow("thresh",thresh)
cv2.waitKey(0)
cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
cv2.drawContours(image, cnts, -1, (0,255,0), 1)
cv2.imshow("contours", image)
cv2.waitKey(0)
for c in cnts:
area = cv2.contourArea(c)
print(area)
if area > 50:
M = cv2.moments(c)
cX = int(M['m10'] / M['m00'])
cY = int(M['m01'] / M['m00'])
#cv2.drawContours(image, [c], -1, (0,255,0), 2) # draw contours on image
(x,y,w,h) = cv2.boundingRect(c) # for each contour get a
bounding rectangle
mask = np.zeros(image.shape[:2], dtype = "uint8") # find
shape of the image dimensions and set up a mask
mask[y: y + h, x: x + w] = 255 # convert region of
interest into white
to_display = cv2.bitwise_and(image,image, mask = mask) # carry
out bitwise and
#cv2.putText(image, 'center', (c))
cv2.imshow("Image", to_display)
cv2.waitKey(0)
this is the screenshot that i am running my code on.
The leftmost screenshot represents the image after applying a threshold to it.
The middle image represents the image i get after drawing the contours.
The last image shows when i am examining each individual contour. The contour covers the line but does not encapsulate the rectangle.
I have a few questions.
1) Is it possible to sieve out the contours for the white rectangles. What alteration do i have to make to my code to be able to achieve this?
2) I am trying to sieve out the unimportant contours eg. the words and I was thinking if i could use the getArea() function to help me with it. The idea is that i would set a minimum contour size to filter out the smaller contours that account for the words.
This is another image that i have tried to identify the "objects" in this screenshots.
I face the same issue here where i cant identify the white rectangles. I am only identifying the borders of the rectangle.
Would appreciate any form of help as I am still new to openCv
Original images before processing:

Answer 1

There is no need to blur. In fact I makes it harder. Simple thresholding works best with hard transitions. The second image is easiest. There are white items on a grayish background. By selecting only very white values the items are selected.
Result:
Code:
# load image
img = cv2.imread("app.png")
# convert to gray
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# crate a mask that hold only white values (above 250)
ret,thresh1 = cv2.threshold(img2,250,255,cv2.THRESH_BINARY)
# find contours in mask
im2, contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# select large contours (menu items only)
for cnt in contours:
print(cv2.contourArea(cnt))
if cv2.contourArea(cnt) > 5000:
# draw a rectangle around the items
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0),3)
#cv2.drawContours(img, [cnt], 0, (0,255,0), 3) #also works, but has issues with letters at the last item
#show image
cv2.imshow("img", img)
#cv2.imshow("mask", thresh) # shows mask
cv2.waitKey(0)
cv2.destroyAllWindows()
The first image is more complex, because it is divided in by a very thin red line. Selecting colors is easier in HSV colorspace. Next red values are used to create a mask, some noise is removed and then contours are detected.
Result:
# load image
img = cv2.imread("app2.png")
# convert to HSV
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
# set lower and upper color limits
lower_val = np.array([0,0,0])
upper_val = np.array([20,50,255])
# Threshold the HSV image
mask = cv2.inRange(hsv, lower_val, upper_val)
# remove noise
kernel = np.ones((1,2),np.uint8)
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
kernel = np.ones((1,5),np.uint8)
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)
# find contours in mask
im2, contours, hierarchy = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# select large contours (menu items only)
for cnt in contours:
print(cv2.contourArea(cnt))
if cv2.contourArea(cnt) > 1000:
# draw a rectangle around the items
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0),3)
#show image
cv2.imshow("img", img)
cv2.imshow("mask", mask)
cv2.waitKey(0)
cv2.destroyAllWindows()