Python Opencv live desktop screen process - opencv

Question

I'm trying to process live screen. There is a game about catching fish. You have to click on fish when it is in circle. I think that I can process my screen with opencv, find fish and click on it with pyautogui.
I did it but problem is program not fast enough to click. By the way game is a mini game in Metin 2 mmorpg. It is like a hack or bot but I just wondering if can I do that.
Here is my code:
import numpy as np
import cv2
from PIL import ImageGrab
import pyautogui
import time
while True:
img=ImageGrab.grab(bbox=(341,208,430,290))
img_np=np.array(img)
#gray=cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
#lower=np.array([57,91,120])
#upper=np.array([65,95,160])
#mask=cv2.inRange(gray,95,130)
#sonuc=cv2.bitwise_and(gray,gray,mask=mask)
#cv2.imshow('frame',mask)
degsk=np.argwhere(img_np==[123,90,57])
if len(degsk)!=0:
#print(degsk)
yerx=341+degsk[int(len(degsk)/2),1]
yery=208+degsk[int(len(degsk)/2),0]
#pyautogui.click(x=yerx, y=yery)
time.sleep(0.8)
if cv2.waitKey(1)&0xFF==ord('q'):
break
cv2.destroyAllWindows()
As you can see, first I tried mask the screen than I realise that it is not necessary so I found BGR value of fish and programed to find it in numpy array than I took value middle in the array and than i used mouse move function. As I said this is not fast enough to catch fish.
So the program is working but delayed for catch fish. How can I make faster this program?
Game Screen Here

Answer 1

Using mss is really fast. Try this:
import time
import cv2
import mss
import numpy
with mss.mss() as sct:
# Part of the screen to capture
monitor = {"top": 40, "left": 0, "width": 800, "height": 640}
while "Screen capturing":
last_time = time.time()
# Get raw pixels from the screen, save it to a Numpy array
img = numpy.array(sct.grab(monitor))
# Display the picture
cv2.imshow("OpenCV/Numpy normal", img)
# Display the picture in grayscale
# cv2.imshow('OpenCV/Numpy grayscale',
# cv2.cvtColor(img, cv2.COLOR_BGRA2GRAY))
print("fps: {}".format(1 / (time.time() - last_time)))
# Press "q" to quit
if cv2.waitKey(25) & 0xFF == ord("q"):
cv2.destroyAllWindows()
break
More information on https://python-mss.readthedocs.io/index.html

Answer 2

I find IPython best for timing things, so if you start IPython and paste in the following code:
from PIL import ImageGrab
img=ImageGrab.grab(bbox=(341,208,430,290))
You can then time a statement with:
%timeit img=ImageGrab.grab(bbox=(341,208,430,290))
and I get this:
552 ms ± 5.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
So, grabbing the screen takes over 500ms, so you are only going to get under 2 frames/second - without even processing it.
If you want to grab the screen faster, I would suggest ffmpeg. I installed it on my iMac running macOS with homebrew using:
brew install ffmpeg
I can then see the list of available video sources and find what I need to record the screen with:
ffmpeg -f avfoundation -list_devices true -i ""
Sample Output
[AVFoundation input device # 0x7fa7dcf05b40] AVFoundation video devices:
[AVFoundation input device # 0x7fa7dcf05b40] [0] FaceTime HD Camera
[AVFoundation input device # 0x7fa7dcf05b40] [1] Capture screen 0 <--- THIS ONE IS THE SCREEN
[AVFoundation input device # 0x7fa7dcf05b40] AVFoundation audio devices:
[AVFoundation input device # 0x7fa7dcf05b40] [0] MacBook Pro Microphone
[AVFoundation input device # 0x7fa7dcf05b40] [1] CalDigit Thunderbolt 3 Audio
So I know I need input 1 for the screen.
So, if I want to record the screen from top-left corner (0,0) at a width of 400px and height of 200px at 20fps for 10s and pass RGBA8888 data to my fishing program, I can do this:
ffmpeg -y -pix_fmt bgr0 -f avfoundation -r 20 -t 10 -i 1 -filter:v "crop=400:200:0:0" -f rawvideo - | ./fish.py
I can now use the following as my fishing program:
#!/usr/bin/env python3
import numpy as np
import pyautogui
import time
import os, sys
# width, height
w, h = 400, 200
# Bytes per frame - assumes bgr0, i.e. 8-bits of blue, 8-bits red. 8-bits green and 8-bits junk
bytesPerFrame = w * h * 4
while True:
img = sys.stdin.buffer.read(bytesPerFrame)
if len(img) != bytesPerFrame:
break
# Process your video here
Keywords: pyautogui, screen grab, screen-grab, screengrab, slow, Mac, macOS, Python, capture, screen capture, ffmpeg, PIL, OpenCV