At this moment I have the VideoImageTrack class I show bellow (adapted from here) that returns an av VideoFrame. The script I show works fine. My problem is that the frame encoding step:
VideoFrame.from_ndarray(image, format="bgr24")
is quite slow.
Is there a gstreamer pipeline that outputs the already encoded frame and iterable with python-opencv read()?
class VideoImageTrack(VideoStreamTrack):
"""
A video stream track that returns a rotating image.
"""
def __init__(self):
super().__init__() # don't forget this!
self.video = cv2.VideoCapture("v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480,framerate=15/1,bitrate=250000 ! appsink")
async def recv(self):
pts, time_base = await self.next_timestamp()
retval, image = self.video.read()
frame = VideoFrame.from_ndarray(image, format="bgr24")
frame.pts = pts
frame.time_base = time_base
return frame
Although I don't know if processing will be faster but you can try to use GStreamer's videoconvert that transcodes video frames between many formats.
Example pipeline:
v4l2src device=/dev/video0 ! videoconvert ! video/x-raw,format=BGR,width=640,height=480,framerate=15/1 ! appsink
Related
I use some python code to decode H264 Video on NVIDIA JETSON NANO everything works fine.
I see some times that I get this warning:
[ WARN:0] global /home/ubuntu/opencv/modules/videoio/src/cap_gstreamer.cpp (961) open
OpenCV | GStreamer warning: Cannot query video position: status=1, value=253, duration=-1
I do not rely sure that this message is the problem but,
When I think to get this warning I get the performance of 7~7.5FPS, sometimes I don't get this warning, and my performance increase to 10.5FPS.
I would be happy to understand the problem I play some pcap video using colasoft player and capture the video in jetson using this python script.
class Video:
def __init__(self, urlName='udp://127.0.0.1:46002'):
print(f"Initialize Vieo, url: {urlName}")
self.pipeline = 'udpsrc port=46002 multicast-group=234.0.0.0 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'
# self.cap = cv2.VideoCapture(urlName, cv2.CAP_FFMPEG)
self.cap = cv2.VideoCapture(self.pipeline, cv2.CAP_GSTREAMER)
if not self.cap.isOpened():
print('VideoCapture not opened')
exit(-1)
def readVideo(self):
"""
This Function read h.264 stream and calculate image info
:return: image, data_image
"""
try:
ret, frame = self.cap.read()
while frame == None:
ret, frame = self.cap.read()
except Exception as e:
pass
shape = frame.shape
data = {"image_width": shape[1], "image_height": shape[0], "channels": shape[2], "size_in_bytes": shape[0] * shape[1] * shape[2], "bits_per_pixel": shape[2] * 8} # data Video
return frame, data
I'm working on a project in which a picamera records to stdout.
I'm adding object detection to the video and need to write these frames to stdout in h264.
So far I've got this:
for frame in self.camera.capture_continuous(rawCapture, format="bgr",use_video_port=True):
#while True:
# grab the raw NumPy array representing the image, then
# initialize the timestamp and occupied/unoccupied text
frame = frame.array
if frame is None:
break
object_detector_frame = self.object_detector.draw_boxes(frame)
sys.stdout.write(object_detector_frame.tostring())
#writer.write(frame)
rawCapture.truncate(0)
CV2's video writer doesn't seem to be able to write bytes straight to stdout.
Thanks in advance.
I'm trying to capture video from an in-build webcam on a laptop (or external USB camera) using opencv, specifically VideoCapture with the DSHOW argument.
I know there is a way to set the resolution and even FPS, however the DirectShow argument for the API returns none when I included it in the code.
For example;
# returns my webcam's stream, but all optional arguments are ignored
camera = cv2.VideoCapture(0)
camera = cv2.VideoCapture(0, cv2.CAP_V4L2)
# returns none and loops infinitely or errors out when *if im.any()*
camera = cv2.VideoCapture(0, cv2.CAP_DSHOW)
This is the code that follows after the above;
# should set resolution, settings are always ignored
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
while(True):
retval, im = camera.read()
if im.any(): # errors out when image is none
cv2.imshow("image", im)
k = cv2.waitKey(33)
if k==27: # Esc key press
print('Resolution: {0}x and {1}y'.format(im.shape[1],im.shape[0]))
print('FPS: {0}'.format(camera.get(cv2.CAP_PROP_FPS)))
break
camera.release()
cv2.destroyAllWindows()
Is the DSHOW the correct API to use and is it the only API to use that can change resolution and FPS of a camera stream using opencv? Or is there something else I'm doing incorrectly?
More details about the system.
Ubuntu 18.04.6
python 3.9.5
opencv-python 4.5.2.52
Thank you in advance for the help!
Regards, Tiz
DSHOW (and MSMF) are windows only.
on linux, use V4L, FFMPEG or GSTREAMER
also, please check the return val of capture.set(),
not all properties/values will be supported on any given machine
I have an RTP/RTSP stream that's running at 25fps, as verified by ffprobe -i <URI>. Also, VLC plays back the RTSP stream at a real-time rate, but doesn't show me the FPS in the Media Information window.
However, when I use OpenCV 4.1.1.26 to retrieve the input stream's frame rate, it is giving me a response of 90000.0.
Question: How can I use OpenCV to probe for the correct frame rate of the RTSP stream? What would cause it to report 90000.0 instead of 25?
Here's my Python function to retrieve the frame rate:
import cv2
vid : cv2.VideoCapture = cv2.VideoCapture('rtsp://192.168.1.10/cam1/mpeg4')
def get_framerate(video: cv2.VideoCapture):
fps = video.get(cv2.CAP_PROP_FPS)
print('FPS is {0}'.format(fps))
get_framerate(vid)
MacOS Catalina
Python 3.7.4
I hope this helps you somehow. It is a simple calculator that takes cont captures and measure the beginning and the ending time. Then with the rule of three, i converted it to fps.
Related to you second question i read here that it could be due to bad installation. Also, you can check that your camera is working properly by printing ret variable. If it is true then you should be able to see the fps, if it is false then you can have an unpredictable result.
cv2.imshow() and key = cv2.waitKey(1) should be commented as it adds ping/delay resulting in bad measurement.
I post this as a comment because i do not have enough reputation points.
img = cv2.VideoCapture('rtsp://192.168.1.10/cam1/mpeg4')
while True:
if cont == 50:
a = datetime.now() - start
b = (a.seconds * 10e6 + a.microseconds)
print((a.seconds * 10e6 + a.microseconds), "fps = ", (50 * 10e6)/ b)
break
ret, frame = img.read()
# Comment for best test
cv2.imshow('fer', frame)
key = cv2.waitKey(1)
if key == ord('q'):
break
cont+=1
img.release()
cv2.destroyAllWindows()`
im using gstreamer 1.2 to feed frames from my IP camera to opencv program
the stream is (640*368 YUVj420p) and i want to convert it to RBG888 to be able to use it in my opencv program
so is there a way to use gstreamer to do that conversion ?
or do i have to do it by myself?
if so please give me the equation that do this conversion
After some trials with gstreamer i decided to do the conversion myself and it worked
First we have to understand the YUVj420p pixel format
As shown in the above image, the Y', U and V components in Y'UV420 are encoded separately in sequential blocks. A Y' value is stored for every pixel, followed by a U value for each 2×2 square block of pixels, and finally a V value for each 2×2 block. Corresponding Y', U and V values are shown using the same color in the diagram above. Read line-by-line as a byte stream from a device, the Y' block would be found at position 0, the U block at position x×y (6×4 = 24 in this example) and the V block at position x×y + (x×y)/4 (here, 6×4 + (6×4)/4 = 30).(copied)
here is the code to do it (python)
This code will show how to inject frame to opencv using gstreamer and make the converstion
import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst
import numpy as np
import cv2
GObject.threads_init()
Gst.init(None)
def YUV_stream2RGB_frame(data):
w=640
h=368
size=w*h
stream=np.fromstring(data,np.uint8) #convert data form string to numpy array
#Y bytes will start form 0 and end in size-1
y=stream[0:size].reshape(h,w) # create the y channel same size as the image
#U bytes will start from size and end at size+size/4 as its size = framesize/4
u=stream[size:(size+(size/4))].reshape((h/2),(w/2))# create the u channel its size=framesize/4
#up-sample the u channel to be the same size as the y channel and frame using pyrUp func in opencv2
u_upsize=cv2.pyrUp(u)
#do the same for v channel
v=stream[(size+(size/4)):].reshape((h/2),(w/2))
v_upsize=cv2.pyrUp(v)
#create the 3-channel frame using cv2.merge func watch for the order
yuv=cv2.merge((y,u_upsize,v_upsize))
#Convert TO RGB format
rgb=cv2.cvtColor(yuv,cv2.cv.CV_YCrCb2RGB)
#show frame
cv2.imshow("show",rgb)
cv2.waitKey(5)
def on_new_buffer(appsink):
sample = appsink.emit('pull-sample')
#get the buffer
buf=sample.get_buffer()
#extract data stream as string
data=buf.extract_dup(0,buf.get_size())
YUV_stream2RGB_frame(data)
return False
def Init():
CLI="rtspsrc name=src location=rtsp://192.168.1.20:554/live/ch01_0 latency=10 !decodebin ! appsink name=sink"
#simplest way to create a pipline
pipline=Gst.parse_launch(CLI)
#getting the sink by its name set in CLI
appsink=pipline.get_by_name("sink")
#setting some important properties of appsnik
appsink.set_property("max-buffers",20) # prevent the app to consume huge part of memory
appsink.set_property('emit-signals',True) #tell sink to emit signals
appsink.set_property('sync',False) #no sync to make decoding as fast as possible
appsink.connect('new-sample', on_new_buffer) #connect signal to callable func
def run():
pipline.set_state(Gst.State.PLAYING)
GObject.MainLoop.run()
Init()
run()
How exactly are you getting the frames from your camera? And how you inject it into your opencv application?
Supposing you get your frames outside of gstreamer you should use a pipeline like:
appsrc caps="video/x-raw, format=I420, width=640, height=368" ! videoconvert ! capsfilter caps="video/x-raw, format=RGB" ! appsink
And then use appsrc to inject the data and use appsink to receive it back. If you are getting your data from camera from http or v4l2 you can replace appsrc with souphttpsrc or v4l2src.