Convert YUVj420p pixel format to RGB888 using gstreamer - opencv

im using gstreamer 1.2 to feed frames from my IP camera to opencv program
the stream is (640*368 YUVj420p) and i want to convert it to RBG888 to be able to use it in my opencv program
so is there a way to use gstreamer to do that conversion ?
or do i have to do it by myself?
if so please give me the equation that do this conversion

After some trials with gstreamer i decided to do the conversion myself and it worked
First we have to understand the YUVj420p pixel format
As shown in the above image, the Y', U and V components in Y'UV420 are encoded separately in sequential blocks. A Y' value is stored for every pixel, followed by a U value for each 2×2 square block of pixels, and finally a V value for each 2×2 block. Corresponding Y', U and V values are shown using the same color in the diagram above. Read line-by-line as a byte stream from a device, the Y' block would be found at position 0, the U block at position x×y (6×4 = 24 in this example) and the V block at position x×y + (x×y)/4 (here, 6×4 + (6×4)/4 = 30).(copied)
here is the code to do it (python)
This code will show how to inject frame to opencv using gstreamer and make the converstion
import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst
import numpy as np
import cv2
GObject.threads_init()
Gst.init(None)
def YUV_stream2RGB_frame(data):
w=640
h=368
size=w*h
stream=np.fromstring(data,np.uint8) #convert data form string to numpy array
#Y bytes will start form 0 and end in size-1
y=stream[0:size].reshape(h,w) # create the y channel same size as the image
#U bytes will start from size and end at size+size/4 as its size = framesize/4
u=stream[size:(size+(size/4))].reshape((h/2),(w/2))# create the u channel its size=framesize/4
#up-sample the u channel to be the same size as the y channel and frame using pyrUp func in opencv2
u_upsize=cv2.pyrUp(u)
#do the same for v channel
v=stream[(size+(size/4)):].reshape((h/2),(w/2))
v_upsize=cv2.pyrUp(v)
#create the 3-channel frame using cv2.merge func watch for the order
yuv=cv2.merge((y,u_upsize,v_upsize))
#Convert TO RGB format
rgb=cv2.cvtColor(yuv,cv2.cv.CV_YCrCb2RGB)
#show frame
cv2.imshow("show",rgb)
cv2.waitKey(5)
def on_new_buffer(appsink):
sample = appsink.emit('pull-sample')
#get the buffer
buf=sample.get_buffer()
#extract data stream as string
data=buf.extract_dup(0,buf.get_size())
YUV_stream2RGB_frame(data)
return False
def Init():
CLI="rtspsrc name=src location=rtsp://192.168.1.20:554/live/ch01_0 latency=10 !decodebin ! appsink name=sink"
#simplest way to create a pipline
pipline=Gst.parse_launch(CLI)
#getting the sink by its name set in CLI
appsink=pipline.get_by_name("sink")
#setting some important properties of appsnik
appsink.set_property("max-buffers",20) # prevent the app to consume huge part of memory
appsink.set_property('emit-signals',True) #tell sink to emit signals
appsink.set_property('sync',False) #no sync to make decoding as fast as possible
appsink.connect('new-sample', on_new_buffer) #connect signal to callable func
def run():
pipline.set_state(Gst.State.PLAYING)
GObject.MainLoop.run()
Init()
run()

How exactly are you getting the frames from your camera? And how you inject it into your opencv application?
Supposing you get your frames outside of gstreamer you should use a pipeline like:
appsrc caps="video/x-raw, format=I420, width=640, height=368" ! videoconvert ! capsfilter caps="video/x-raw, format=RGB" ! appsink
And then use appsrc to inject the data and use appsink to receive it back. If you are getting your data from camera from http or v4l2 you can replace appsrc with souphttpsrc or v4l2src.

Related

Open CV write h264 encoded frames to stdout

I'm working on a project in which a picamera records to stdout.
I'm adding object detection to the video and need to write these frames to stdout in h264.
So far I've got this:
for frame in self.camera.capture_continuous(rawCapture, format="bgr",use_video_port=True):
#while True:
# grab the raw NumPy array representing the image, then
# initialize the timestamp and occupied/unoccupied text
frame = frame.array
if frame is None:
break
object_detector_frame = self.object_detector.draw_boxes(frame)
sys.stdout.write(object_detector_frame.tostring())
#writer.write(frame)
rawCapture.truncate(0)
CV2's video writer doesn't seem to be able to write bytes straight to stdout.
Thanks in advance.

OpenCV 4.1.1.26 reports 90000.0 fps for a 25fps RTSP stream

I have an RTP/RTSP stream that's running at 25fps, as verified by ffprobe -i <URI>. Also, VLC plays back the RTSP stream at a real-time rate, but doesn't show me the FPS in the Media Information window.
However, when I use OpenCV 4.1.1.26 to retrieve the input stream's frame rate, it is giving me a response of 90000.0.
Question: How can I use OpenCV to probe for the correct frame rate of the RTSP stream? What would cause it to report 90000.0 instead of 25?
Here's my Python function to retrieve the frame rate:
import cv2
vid : cv2.VideoCapture = cv2.VideoCapture('rtsp://192.168.1.10/cam1/mpeg4')
def get_framerate(video: cv2.VideoCapture):
fps = video.get(cv2.CAP_PROP_FPS)
print('FPS is {0}'.format(fps))
get_framerate(vid)
MacOS Catalina
Python 3.7.4
I hope this helps you somehow. It is a simple calculator that takes cont captures and measure the beginning and the ending time. Then with the rule of three, i converted it to fps.
Related to you second question i read here that it could be due to bad installation. Also, you can check that your camera is working properly by printing ret variable. If it is true then you should be able to see the fps, if it is false then you can have an unpredictable result.
cv2.imshow() and key = cv2.waitKey(1) should be commented as it adds ping/delay resulting in bad measurement.
I post this as a comment because i do not have enough reputation points.
img = cv2.VideoCapture('rtsp://192.168.1.10/cam1/mpeg4')
while True:
if cont == 50:
a = datetime.now() - start
b = (a.seconds * 10e6 + a.microseconds)
print((a.seconds * 10e6 + a.microseconds), "fps = ", (50 * 10e6)/ b)
break
ret, frame = img.read()
# Comment for best test
cv2.imshow('fer', frame)
key = cv2.waitKey(1)
if key == ord('q'):
break
cont+=1
img.release()
cv2.destroyAllWindows()`

How do I configure while loop to use picamera?

So I want to read individual frames of picamera using while loop. Here is what I found for doing it using for loop:
# import the necessary packages
from picamera.array import PiRGBArray
from picamera import PiCamera
import time
import cv2
# initialize the camera and grab a reference to the raw camera capture
camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(640, 480))
# allow the camera to warmup
time.sleep(0.1)
# capture frames from the camera
for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
# grab the raw NumPy array representing the image, then initialize the timestamp
# and occupied/unoccupied text
image = frame.array
# show the frame
cv2.imshow("Frame", image)
key = cv2.waitKey(1) & 0xFF
# clear the stream in preparation for the next frame
rawCapture.truncate(0)
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
cv2.destroyAllWindows()
Now when I use the above code, I get the video feed but I am planning to do the same using a while loop. Going from the same logic I added a while loop something like this:
while True: frame1=camera.capture_continious(rawCapture,format="bgr",use_video_port=True)
image1 = frame1.array
# show the frame
cv2.imshow("Frame1", image1)
# clear the stream in preparation for the next frame
rawCapture.truncate(0)
But still I am getting an error as frame1 is a generator and doesn't contain such attributes while the same code is running well with for loop. What modifications can I make?
The function capture_continuous() returns an infinite iterator of images captured continuously from the camera. It does not return a single image. That's is why it does work with a for loop.
In your while loop you should use the capture() function, which does return an image.
You can (and should ;) ) read more about it in this documentation

It s there any way to acces the frames taken with opencv for encription?

I try to make an encrypted video conference, so the data will be sent UDP mode, I accessed the webcam with openCV in python, it is there any whay to acces the data from webcam as bits so that I can encript-transfer-decript and show the immage on the desired station? Or what other library(python/java/c/c++) can help me wth taking data from my webcam?
I'm using OpenCV 3.3 for Python(3.5) on Ubuntu 16.04.
For you question, to encrypt the frame taken from video and transform by TCP/UDP:
On Side A:
(1) use `cap.read` to take frame form `videoCapture` as `np.array`
(2) use `np.tobytes` to convert from `np.array` to `bytes`
(3) do encryption on bytes(can be switched with step 2).
(4) transfer `bytes` using TCP/UDP
On Side B:
(1) receive from A
(2) do decryption on the received bytes
(3) use `np.frombuffer` to convert from `bytes` to `np.array`, then `reshape`. you get the data.
A sample code snippet:
On Side A
cap = cv2.VideoCapture(0)
assert cap.isOpened()
## (1) read
ret, frame = cap.read()
sz = frame.shape
## (2) numpy to bytes
frame_bytes = frame.tobytes()
print(type(frame_bytes))
# <class 'bytes'>
## (3) encode
# ...
## (4) transfer
# ...
cap.release()
On Side B
## (1) receive
## (2) decode
## (3) frombuffer and reshape to the same size on Side A
frame_frombytes = np.frombuffer(frame_bytes, dtype=np.uint8).reshape(sz)
print(type(frame_frombytes))
## <class 'numpy.ndarray'>
Related answers:
(1) Convert str to numpy.ndarray
(2) Can't write video by opencv in Python
(3) How to transfer cv::VideoCapture frames through socket in client-server model (OpenCV C++)?

Simultneous depth and video from kinect

I want to get both depth and video from streams from the kinect to my opencv code. I am working in Linux. I have installed libfreenect module for depth. However, there is only one device listed in /dev/. Now, when I connect the Kinect to my pc and run
camorama -d /dev/video0
I get the depth map. Then, I access the device using videocapture in opencv and I get the rgb video. Now, if I again run the camorama command, I get the rgb video this time. I can't figure out what's happening. I basically want both the stream in my opencv code. Please help.
Run this python script:
import freenect
import cv2
import numpy as np
from functions import *
def nothing(x):
pass
kernel = np.ones((5, 5), np.uint8)
def pretty_depth(depth):
np.clip(depth, 0, 2**10 - 1, depth)
depth >>= 2
depth = depth.astype(np.uint8)
return depth
while 1:
orig = freenect.sync_get_video()[0]
orig = cv2.cvtColor(orig,cv2.COLOR_BGR2RGB)
dst = pretty_depth(freenect.sync_get_depth()[0])#input from kinect
cv2.imshow('Disparity', dst)
cv2.imshow('RGB',orig)
if cv2.waitKey(1) & 0xFF == ord('b'):
break

Resources