Line following robot in outdoor condition with intel sealsense d415 - opencv

I am working on making a line following robot using Opencv and am using nvidia realsense d415 depth camera for image capturing.I am using cv2.findcontours and taking the max contour as the line and then working on that. the problem is along with detecting the main line , the code is also detecting random contours, like when I capture empty space on ground, it considers the whole frame as one contour. It is also picking up random objects as contour. Is there a way to detect just the black line? and no other object as contour?
Code:
cap = cv2.VideoCapture(2)
ret,frame = cap.read()
frame = frame[0:800,0:1800]
value = 80 #whatever value you want to add
cv2.add(frame[:,:,2], value, frame[:,:,2])
img = cv2.cvtColor(frame, cv2.COLOR_HSV2BGR)
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img = cv2.GaussianBlur(img,(5,5),0)
_,threshold = cv2.threshold(img,60,255,cv2.THRESH_BINARY_INV)
mask = cv2.erode(threshold, None, iterations=2)
mask = cv2.dilate(mask, None, iterations=2)
contours, hierarchy =
cv2.findContours(mask.copy(),1,cv2.CHAIN_APPROX_NONE)
max_c = max(contours, key=cv2.contourArea)
cv2.drawContours(img, max_c, -1, (0,255,0), 3)
Image with object

Related

filtering out anything besides white in opencv

I am currently developing a robot that stays within the lines and avoids circles on the ground of the same color as the lines. I want to print all the coordinates(x,y) to the terminal to get started to make sure I am detecting the correct pixels. I wrote a program that I thought filtered out anything besides white and prints out the x,y coordinates of the white pixels, but it doesn't seem to be working. I passed through a completely black image and a completely white image and received the same coordinates. Can anyone tell me what I am doing wrong?
import cv2
import numpy as np
#cap = cv2.VideoCapture(1)
frame = cv2.imread("black.png")
while(True):
#_, frame = cap.read()
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
# define range of white color in HSV
lower_white = np.array([0, 0, 0], dtype=np.uint8)
upper_white = np.array([0, 0, 255], dtype=np.uint8)
# Threshold the HSV image to get only white colors
mask = cv2.inRange(hsv, lower_white, upper_white)
# Bitwise-AND mask and original image
res = cv2.bitwise_and(frame,frame, mask= mask)
coord = cv2.findNonZero(mask)
print(coord)
cv2.imshow('frame',frame)
cv2.imshow('mask',mask)
cv2.imshow('res',res)
k = cv2.waitKey(5) & 0xFF
if k == 27:
break
cv2.destroyAllWindows()
Black image
White image
Test image

Determine flow meter reading from image

currently I'm working on a project to identify the rotameter's reading. My main task is to located the rotameter from the input image, then identify the line mark from the rotameter as well as the floater. Then, I need to find out the reading of the meter based on the position of the floater. There are a few challenges faced, the glare on the meter due to illumination and the angle of the image taken. Currently, I tried with CLAHE followed by Canny edge detection. However, I was only able to get the line mark at the area without light reflection. I had also tried Inpaint + CLAHE to remove the glare part and Gaussian blurring to reduce the image noises. But most on the line marks are missing. I'm still new in computer vision and currently using Python + OpenCV. Any good suggestions?
Source image
Here's the code I still trying on for Inpaint + CLAHE, and the outcome is attached.
Outcome: Inpaint + Clahe
img = cv2.imread('IMG-20210630-WA0016.jpeg')
resized = cv2.resize(img, (540, 960))
crp = resized[0:960, 150:360]
# convert image from RGB to HSV
img_hsv = cv2.cvtColor(crp, cv2.COLOR_RGB2HSV)
# Histogram equalisation on the V-channel
img_hsv[:, :, 2] = cv2.equalizeHist(img_hsv[:, :, 2])
# convert image back from HSV to RGB
image = cv2.cvtColor(img_hsv, cv2.COLOR_HSV2RGB)
gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
# Create a CLAHE object (Arguments are optional).
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(4,4))
cl1 = clahe.apply(gray)
cv2.imshow('Clahe', cl1)
# mask = cv2.threshold(cl1, 180, 185, cv2.THRESH_BINARY)[1]
mask = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
# use mask with input to do inpainting
inpaint = cv2.inpaint(cl1, mask, 21, cv2.INPAINT_NS)
# display it
cv2.imshow("IMAGE", crp)
cv2.imshow("GRAY", gray)
result1 = cv2.GaussianBlur(inpaint, (1, 1), 1, cv2.BORDER_DEFAULT)
canny = cv2.Canny(result1, 100, 255)
cv2.imshow("Canny", canny)
key = cv2.waitKey(0)
if key == 27:
cv2.destroyAllWindows()
For this, I just tried with CLAHE, the line marks look clearer but the position of the floater are missing.
Outcome 2: Clahe
img = cv2.imread('IMG-20210630-WA0016.jpeg')
resized = cv2.resize(img, (540, 960))
crp = resized[0:960, 150:360]
gray = cv2.cvtColor(crp,cv2.COLOR_BGR2GRAY)
# Create a CLAHE object (Arguments are optional).
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(4,4))
cl1 = clahe.apply(gray)
result1 = cv2.GaussianBlur(cl1, (3, 3), 1, cv2.BORDER_DEFAULT)
canny = cv2.Canny(result1, 150, 255)
cv2.imshow("Canny", canny)
cv2.imshow("Source", crp)
cv2.imshow('Clahe', cl1)
key = cv2.waitKey(0)
if key == 27:
cv2.destroyAllWindows()

How to detect computer/laptop's screen in an image?

I am trying to detect the screen of the monitor either computer or laptop.
Original image
Points of the image
First picture is the original image, second image marked with points of the rectangular area I want to detect in an image.
I have tried to get the screen by using cv2.findContours with opencv by this tutorial but it didn't help. In tutorial he had the frontal picture of the screen but I have angular images mostly(videos) so it crashes to grab good contours and define the screen.
Code I used to find the screen:
import numpy as np
import imutils
import cv2
from PIL import Image
args = {
'query': '/Users/PC/Desktop/screendetect/mm.jpeg'
}
class dotdict(dict):
def __getattr__(self, name):
return self[name]
args = dotdict(args)
# load the query image, compute the ratio of the old height
# to the new height, clone it, and resize it
image = cv2.imread(args["query"])
ratio = image.shape[0] / 300.0
orig = image.copy()
image = imutils.resize(image, height = 300)
# convert the image to grayscale, blur it, and find edges
# in the image
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.bilateralFilter(gray, 11, 17, 17)
edged = cv2.Canny(gray, 30, 200)
# find contours
cnts = cv2.findContours(edged.copy(), cv2.RETR_TREE,
cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:100]
screenCnt = None
# loop over our contours
for c in cnts:
# approximate the contour
peri = cv2.arcLength(c, True)
approx = cv2.approxPolyDP(c, 0.03 * peri, True)
# if our approximated contour has four points, then
# we can assume that we have found our screen
if len(approx):
screenCnt = approx
break
# draw a rectangle around the screen
orig = image.copy()
d = cv2.drawContours(image, [screenCnt], -1, (0, 255, 0), 1)
cv2.imwrite("/Users/PC/Desktop/screendetect/test_good.jpg", d)
# cv2.waitKey(0)
In other test images there will be only one monitor so I need to find only one screen. What is the best approach to tackle this kind of problem?

openCV problem with detecting contours of shapes fully

I am doing this university project where i try to detect UI elements on screenshots of Android applications using openCV. I am not expecting a 100 percent accuracy for this detection of UI elements.
This is my code below. I convert the image to gray scale, apply Gaussian blur and then use adaptive threshold to convert the image to binary. After which i use the find contours method.
ap = argparse.ArgumentParser()
ap.add_argument("-i","--image", help = "path to an image", required =
True)
args = vars(ap.parse_args())
image = cv2.imread(args["image"])
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imshow("gray",gray)
cv2.waitKey(0)
blurred = cv2.GaussianBlur(gray, (5,5), 0)
thresh = cv2.adaptiveThreshold(blurred, 255,
cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 11, 4)
cv2.imshow("thresh",thresh)
cv2.waitKey(0)
cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
cv2.drawContours(image, cnts, -1, (0,255,0), 1)
cv2.imshow("contours", image)
cv2.waitKey(0)
for c in cnts:
area = cv2.contourArea(c)
print(area)
if area > 50:
M = cv2.moments(c)
cX = int(M['m10'] / M['m00'])
cY = int(M['m01'] / M['m00'])
#cv2.drawContours(image, [c], -1, (0,255,0), 2) # draw contours on image
(x,y,w,h) = cv2.boundingRect(c) # for each contour get a
bounding rectangle
mask = np.zeros(image.shape[:2], dtype = "uint8") # find
shape of the image dimensions and set up a mask
mask[y: y + h, x: x + w] = 255 # convert region of
interest into white
to_display = cv2.bitwise_and(image,image, mask = mask) # carry
out bitwise and
#cv2.putText(image, 'center', (c))
cv2.imshow("Image", to_display)
cv2.waitKey(0)
this is the screenshot that i am running my code on.
The leftmost screenshot represents the image after applying a threshold to it.
The middle image represents the image i get after drawing the contours.
The last image shows when i am examining each individual contour. The contour covers the line but does not encapsulate the rectangle.
I have a few questions.
1) Is it possible to sieve out the contours for the white rectangles. What alteration do i have to make to my code to be able to achieve this?
2) I am trying to sieve out the unimportant contours eg. the words and I was thinking if i could use the getArea() function to help me with it. The idea is that i would set a minimum contour size to filter out the smaller contours that account for the words.
This is another image that i have tried to identify the "objects" in this screenshots.
I face the same issue here where i cant identify the white rectangles. I am only identifying the borders of the rectangle.
Would appreciate any form of help as I am still new to openCv
Original images before processing:
There is no need to blur. In fact I makes it harder. Simple thresholding works best with hard transitions. The second image is easiest. There are white items on a grayish background. By selecting only very white values the items are selected.
Result:
Code:
# load image
img = cv2.imread("app.png")
# convert to gray
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# crate a mask that hold only white values (above 250)
ret,thresh1 = cv2.threshold(img2,250,255,cv2.THRESH_BINARY)
# find contours in mask
im2, contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# select large contours (menu items only)
for cnt in contours:
print(cv2.contourArea(cnt))
if cv2.contourArea(cnt) > 5000:
# draw a rectangle around the items
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0),3)
#cv2.drawContours(img, [cnt], 0, (0,255,0), 3) #also works, but has issues with letters at the last item
#show image
cv2.imshow("img", img)
#cv2.imshow("mask", thresh) # shows mask
cv2.waitKey(0)
cv2.destroyAllWindows()
The first image is more complex, because it is divided in by a very thin red line. Selecting colors is easier in HSV colorspace. Next red values are used to create a mask, some noise is removed and then contours are detected.
Result:
# load image
img = cv2.imread("app2.png")
# convert to HSV
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
# set lower and upper color limits
lower_val = np.array([0,0,0])
upper_val = np.array([20,50,255])
# Threshold the HSV image
mask = cv2.inRange(hsv, lower_val, upper_val)
# remove noise
kernel = np.ones((1,2),np.uint8)
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
kernel = np.ones((1,5),np.uint8)
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)
# find contours in mask
im2, contours, hierarchy = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# select large contours (menu items only)
for cnt in contours:
print(cv2.contourArea(cnt))
if cv2.contourArea(cnt) > 1000:
# draw a rectangle around the items
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0),3)
#show image
cv2.imshow("img", img)
cv2.imshow("mask", mask)
cv2.waitKey(0)
cv2.destroyAllWindows()

OpenCV - Extracting lines on a graph

I would like to create a program that is able to extract lines from a graph.
For example, if a graph like this is inputted, I would just want the red line to be outputted.
Below I have tried to do this using a hough line transformation, however, I do not get very promising results.
import cv2
import numpy as np
graph_img = cv2.imread("/Users/2020shatgiskessell/Desktop/Graph1.png")
gray = cv2.cvtColor(graph_img, cv2.COLOR_BGR2GRAY)
kernel_size = 5
#grayscale image
blur_gray = cv2.GaussianBlur(gray,(kernel_size, kernel_size),0)
#Canny edge detecion
edges = cv2.Canny(blur_gray, 50, 150)
#Hough Lines Transformation
#distance resoltion of hough grid (pixels)
rho = 1
#angular resolution of hough grid (radians)
theta = np.pi/180
#minimum number of votes
threshold = 15
#play around with these
min_line_length = 25
max_line_gap = 20
#make new image
line_image = np.copy(graph_img)
#returns array of lines
lines = cv2.HoughLinesP(edges, rho, theta, threshold, np.array([]),
min_line_length, max_line_gap)
for line in lines:
for x1,y1,x2,y2 in line:
cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0),2)
lines_edges = cv2.addWeighted(graph_img, 0.8, line_image, 1, 0)
cv2.imshow("denoised image",edges)
if cv2.waitKey(0) & 0xff == 27:
cv2.destroyAllWindows()
This produces the output image below, which does not accurately recognize the graph line. How might I go about doing this?
Note: For now, I am not concerned about the graph titles or any other text.
I would also like the code to work for other graph images aswell, such as:
etc.
If the graph does not have many noises around it (like your example) I would suggest to threshold your image with Otsu threshold instead of looking for edges . Then you simply search the contours, select the biggest one (graph) and draw it on a blank mask. After that you can perform a bitwise operation on image with the mask and you will get a black image with the graph. If you like the white background better, then simply change all black pixels to white. Steps are written in the example. Hope it helps a bit. Cheers!
Example:
import numpy as np
import cv2
# Read the image and create a blank mask
img = cv2.imread('graph.png')
h,w = img.shape[:2]
mask = np.zeros((h,w), np.uint8)
# Transform to gray colorspace and threshold the image
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
_, thresh = cv2.threshold(gray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
# Search for contours and select the biggest one and draw it on mask
_, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
cnt = max(contours, key=cv2.contourArea)
cv2.drawContours(mask, [cnt], 0, 255, -1)
# Perform a bitwise operation
res = cv2.bitwise_and(img, img, mask=mask)
# Convert black pixels back to white
black = np.where(res==0)
res[black[0], black[1], :] = [255, 255, 255]
# Display the image
cv2.imshow('img', res)
cv2.waitKey(0)
cv2.destroyAllWindows()
Result:
EDIT:
For noisier pictures you could try this code. Note that different graphs have different noises and may not work on every graph image since the denoisiation process would be specific in every case. For different noises you can use different ways to denoise it, for example histogram equalization, eroding, blurring etc. This code works well for all 3 graphs. Steps are written in comments. Hope it helps. Cheers!
import numpy as np
import cv2
# Read the image and create a blank mask
img = cv2.imread('graph.png')
h,w = img.shape[:2]
mask = np.zeros((h,w), np.uint8)
# Transform to gray colorspace and threshold the image
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
_, thresh = cv2.threshold(gray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
# Perform opening on the thresholded image (erosion followed by dilation)
kernel = np.ones((2,2),np.uint8)
opening = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel)
# Search for contours and select the biggest one and draw it on mask
_, contours, hierarchy = cv2.findContours(opening,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
cnt = max(contours, key=cv2.contourArea)
cv2.drawContours(mask, [cnt], 0, 255, -1)
# Perform a bitwise operation
res = cv2.bitwise_and(img, img, mask=mask)
# Threshold the image again
gray = cv2.cvtColor(res,cv2.COLOR_BGR2GRAY)
_, thresh = cv2.threshold(gray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
# Find all non white pixels
non_zero = cv2.findNonZero(thresh)
# Transform all other pixels in non_white to white
for i in range(0, len(non_zero)):
first_x = non_zero[i][0][0]
first_y = non_zero[i][0][1]
first = res[first_y, first_x]
res[first_y, first_x] = 255
# Display the image
cv2.imshow('img', res)
cv2.waitKey(0)
cv2.destroyAllWindows()
Result:

Resources