My FPS fall drastically when recording a video - opencv

I'm using OpenCV to record video for a Full HD Camera Running at 30 FPS. I know the FPS of the camera because I measure it by counting the number of valid frames (this also coincides with the specifications). I do this by using a 5 ms second timer. Here is the code that I run each 5 ms.
cv::Mat frame;
if (capture.read(frame)){
showCurrentFrame(frame);
fpsCounter.newFrame();
ui->labelVideo->setText("Video (" + QString::number(frame.cols) + "x" + QString::number(frame.rows) + ")");
ui->labFPS->setText("FPS: " + QString::number(fpsCounter.getFPS()));
if (isRecording){
recorder << frame;
fpsRecordCounter++;
}
}
I set the FPS of the recorded video to 30 when I press the "Start Recording Button"
recorder.open(currentVideoFile.toStdString(),
VIDEO_CODEC_FOURCC, // Tells it to record MJPEG
VIDEO_REC_FPS, // Define with 30
frameSize)){ // 1920x1080
I've developed my Program in my workstation which runs Centos 7 using Qt and OpenCV 2.4.5. When I record in the desktop PC the FPS show consistently to be about 30.
However, this needs to record from a moving car. So I copy-pasted the code AS IS into my laptop and compiled it with zero issues.
My laptop uses Debian Testing and OpenCV 2.4.9. It is HERE that the slowdown is observed.
Since I'm using Qt I need to process the cv::Mat in order to display it. I do this in the showCurrentFrame function.
If I deactivate this function when recording I get maybe 23 FPS (and can't see what I'm recording)
If I leave it as is in the code above I get about 16~17 FPS.
The first thing that I thought was that my computer was not powerful enough, but it shouldn't be. This is the model:
https://www.asus.com/Notebooks/ASUS_VivoBook_S550CA/specifications/
It's the I5 variant with 500 Gb of HDD.
So I'm at a loss. Is it some sort of bug on OpenCV that was introduced in newer openCV versions or is it simply that my laptop is not powerful enough?

Related

Opening opencv cap for webcamera for extended period of time

i have a django web application hosted locally on a desktop workstation that is supposed to retrieve a video feed/video frames from a webcamera connected via usb. What then happens is that the frame would used as input into an object detection model to count some stuff, but since the objects are small, i need to retrieve the frame at a higher resolution (720, 1280) instead of the default resolution. after the counting is done, i stop reading frames from the web camera - here is the sequence:
Press button in web application to start retrieving video from webcamera
create new opencv cap, since i need a higher resolution, i have to specify the resolution
cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)
while loop that include frame retrieval, counting logic and visualization of the frame with inference results using opencv cv2.imshow()
stop retrieving video frames (cv2.destroyAllWindows() and break out of while loop) after counting is done, finally calling cap.release()
the problem with this sequence is that the video stream takes a few seconds (about 3 seconds) to load (this means that after clicking the button in step 1, i only see the video stream in step 3 after about 3 seconds which is quite slow, and there is an autofocus that takes about 1 second after the video stream pops up)
i am considering this new sequence which does not involve the release of the cap. i found that the video stream takes much faster to load and there is no autofocus each time the video stream is loaded:
when starting the django server, create the opencv cap and specify the required resolution (meaning this is only done once)
cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)
while loop that include frame retrieval, counting logic and visualization of the frame with inference results using opencv cv2.imshow()
stop retrieving video frames (break out of while loop WITHOUT calling cap.release()) after counting is done
Would the second flow result in any problems or damage to the webcamera, especially if the cap is opened for a long time (like weeks or even months).

OpenCV how to grab the real time frame and not the next frame?

I realized that when I use OpenCV to grab videos (cv2.VideoCapture('rtsp://...')) in a rtsp url, actually I am getting everyframe of the stream and not the real time frame.
Example: If a video has 30fps and 10 seconds long, if I get the first frame and wait 1 second to get the next, I get the frame number 2 and not the real time frame (it should be frame number 30 or 31).
I am worried about these because if my code take a little longer to do the video processing (deep learning convolutions), the result will always be delivered later and not in real time.
Any ideas how can I manage to always get the current frame when I capture from rtsp?
Thanks!
This is not about the code. Many IP cameras gives you encoded output(H.265/H.264).
When you used VideoCapture() , the output data of the camera is decoded by the CPU. Getting delay as you mentioned also such as between 1 sec and 2 sec is normal.
What can be done to make it faster:
If you have a GPU hardware, you can decode the data via on it. This
will give you really good results(according to experiences by using
latest version of NVIDIA gpus: you will get almost 25 milisecond
delay) To achieve that on your code, you need:
CUDA installation
CUDA enabled OpenCV installation
VideoReader class of OpenCV
You can use VideoCapture() with FFMPEG flag, FFMPEG has advanced methods to decode encoded data and this will give you probably most faster output which you can get with your CPU. But this will not decrease time much.

Flutter Camera bad quality

i got the Flutter Camera working with a preview and all that. But the quality of the recorded video is way to bad. I get 1MB for 10 seconds. The resolution is fine (1080 x 1440), but i think the bitrate is way to low somehow.
I looked into the CameraPlugin:
https://github.com/flutter/plugins/blob/master/packages/camera/android/src/main/java/io/flutter/plugins/camera/CameraPlugin.java
mediaRecorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264);
mediaRecorder.setVideoEncodingBitRate(1024 * 1000);
mediaRecorder.setAudioSamplingRate(16000);
Is this the normal configuration and does it work for you guys?
I got a Oneplus 2 and normal camera is taking better videos.
I'm not used to the flutter Method Channel so i can't create my own CustomCameraPlugin and change the important values.
Maybe there is a whole different approach.
Let me know
Greetings Markus
Changing mediaRecorder.setVideoEncodingBitRate from 1024 * 1000 to 3000000 for worked for me for better quality.
https://github.com/flutter/plugins/blob/master/packages/camera/android/src/main/java/io/flutter/plugins/camera/CameraPlugin.java

Beagle Bone Black Audio Cape rev B synchronization issues

Basically the audio cape is working. Except for one strange phenomena that mistifies me. I will try to explain.
When I play a .wav file for example speaker-test -t vaw -> if lucky I hear Front Left - Front right as one expects. But 9 out of 10, I hear white noise with the audio front left front right very faint in the background or at another time the sound is simply distorted. The same happens when I play a file with aplay or mplayer.
So when I am lucky, or timing with respect to system clock is in sync I hear the audio clearly, if out of sync it might me white noise or distorted playback.
I have google extensively and have not found any solution. So I hope one of you guys knows whats happening here. It has to be something low level.
I'm quite a newby in this matter but according to this: Troubleshooting Linux Sound all seams to work ok.
These are my system parameters and settings: root#beaglebone:~# lsb_release -a Distributor ID: Angstrom Description: Angstrom GNU/Linux v2012.12 (Core edition) Release: v2012.12 Codename: Core edition
root#beaglebone:~# cat /sys/devices/bone_capemgr*/slots 0: 54:PF---
1: 55:PF---
2: 56:P---L CBB-Relay,00A0,Logic_Supply,CBB-Relay
3: 57:PF---
4: ff:P-O-L Bone-LT-eMMC-2G,00A0,Texas Instrument,BB-BONE-EMMC-2G
5: ff:P-O-- Bone-Black-HDMI,00A0,Texas Instrument,BB-BONELT-HDMI
6: ff:P-O-L Bone-Black-HDMIN,00A0,Texas Instrument,BB-BONELT-HDMIN
7: ff:P-O-L Override Board Name,00A0,Override Manuf,BB-BONE-AUDI-02
root#beaglebone:~# speaker-test -t wav
speaker-test 1.0.25
Playback device is default Stream parameters are 48000Hz, S16_LE, 1 channels WAV file(s) Rate set to 48000Hz (requested 48000Hz) Buffer size range from 128 to 32768 Period size range from 8 to 2048 Using max buffer size 32768 Periods = 4 was set period_size = 2048 was set buffer_size = 32768
0 - Front Left
Time per period = 0.641097
0 - Front Left
root#beaglebone:~# mplayer AxelF.wav MPlayer2 2.0-379-ge3f5043 (C) 2000-2011 MPlayer Team 162 audio & 361 video codecs
Playing AxelF.wav. Detected file format: WAV format (libavformat) [wav # 0xb6082780]max_analyze_duration reached [lavf] stream 0: audio (pcm_s16le), -aid 0 Load subtitles in .
==============================================================[edit]
Forced audio codec: mad Opening audio decoder: [pcm] Uncompressed PCM audio decoder AUDIO: 44100 Hz, 2 ch, s16le, 1411.2 kbit/100.00% (ratio: 176400->176400) Selected audio codec: [pcm] afm: pcm (Uncompressed PCM)
==============================================================[edit]
AO: [alsa] 44100Hz 2ch s16le (2 bytes per sample) Video: no video Starting playback... A: 1.6 (01.6) of 15.9 (15.8) 0.3%
MPlayer interrupted by signal 2 in module: unknown
Exiting... (Quit)
I can shed some light on what is causing the artifacts that you experience. I am sorry I do not yet have a countermeasure - I am struggling with the same problem. You describe the perceptible consequences pretty accurately.
Sound data travels from the ARM System on Chip to the Audio Codec on the audio cape using the I2S bus. I2S is a serial protocol, it sends one bit at a time, starting each sample with the most significant bit, then sending all bits down to the least significant bit. After the least significant bit of one sample is sent, the most significant bit of the sample on the next audio channel is sent. To be able to interpret the bit stream, the receiving audio codec needs to know when a new sound sample starts with its most significant bit, and also, to which channel each sound sample belongs. For this purpose, the "Word Select" (WS) signal is part of I2S and changes its value to indicate the start of the sound sample and also identifies the channel, see this I2S timing diagram for a better understanding of the concept.
What you and I perceive on our not-quite-working audio capes can be fully explained by the bit stream being interpreted out-of-step by the audio codec:
When you hear loud noise and the target signal soft in the background, then one or more of the least significant bits of the preceding sample are interpreted as the most significant bits of the current sample. The more bits are shifted, the softer the target signal, until you might only perceive noise when (this is a guess!) about 4 bits are shifted.
When the shift is in the other direction, i.e. most significant bit of the current sample was interpreted as the least significant bit of the preceding sample, then what you hear will sound correct for soft parts of the signal, i.e. when the most significant bit is not actually used (this is a simplification, see below). For louder parts of the signal, e.g. drum beats, you will perceive the missing most significant bit as distortion. Of course, the distortion gets worse and starts at softer levels as more bits are shifted in this direction.
In the above paragraph, the most significant will change with the sign of the data, so the statement that it is not actually used is valid only insofar as the most significant bit will have the same value as the next most significant bit for soft sounds. See Two's Complement for an introduction how negative integers are represented in computers.
I am not sure, where the corruption occurs. It could be that the WS signal is not correctly interpreted by the Audio Codec on the cape, or the WS signal is not correctly sent by the ARM System-on-Chip, or the bit shift might happen already inside the ARM CPU, e.g. in the Alsa driver.

Lossless avi encoding on linux

I am trying to write video using opencv. It is important for me to do this precisely - so it has to be a lossless codec. I am working with OpenCV 2.4.1 on Ubuntu 12.04
Previously, I was using the fourcc code 0. This gave me the exact result I wanted, and I was able to recover the images perfectly.
I am not sure what happened, but as of a recent update (around Jul 20th 2012), something went wrong and I am no longer able to write files with this fourcc code. I really don't remember what it was, but it could have come from doing an update, removing some software from my software center, and some other things I did during general cleaning...
When I check an older file with mediainfo (http://www.fourcc.org/identifier/) I see the following result:
Complete name : oldsample.avi
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 1.07 GiB
Duration : 41s 467ms
Overall bit rate : 221 Mbps
Writing application : Lavf53.5.0
Video
ID : 0
Format : RGB
Codec ID : 0x00000000
Codec ID/Info : Basic Windows bitmap format. 1, 4 and 8 bpp versions are palettised. 16, 24 and 32bpp contain raw RGB samples
Duration : 41s 467ms
Bit rate : 221 Mbps
Width : 640 pixels
Height : 4294966 816 pixels
Display aspect ratio : 0.000
Frame rate : 30.000 fps
Bit depth : 8 bits
Stream size : 1.07 GiB (100%)
Now, I see that when I write using the 0 fourcc codec, the program actually defaults to the i420 codec. Here is the output from one of the files I try to write now:
Complete name : newsample.avi
Format : AVI
Format/Info : Audio Video Interleave
File size : 73.0 MiB
Duration : 5s 533ms
Overall bit rate : 111 Mbps
Writing application : Lavf54.6.100
Video
ID : 0
Format : YUV
Codec ID : I420
Codec ID/Info : 8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes.
Duration : 5s 533ms
Bit rate : 111 Mbps
Width : 640 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 30.000 fps
Compression mode : Lossless
Bits/(Pixel*Frame) : 12.000
Stream size : 72.9 MiB (100%)
This format, and other formats I try to use (like huffyuv HFYU), do not work for me because I end up with effects like this http://imgur.com/a/0OC4y - you see the bright artifacts coming in due to what I assume is either lossy compression or chroma subsampling in the case of HFYU which is supposed to be lossless. What you are looking at is the red channel from one of my videos. The perceptual effect is negligible when you look at all 3 channels simultaneously but it is essential that I reconstruct the images exactly.
Furthermore, while I am able to play my old files in media players like vlc, I suddenly find them to be completely incompatible with opencv. When I try to open the older files with a videocapture, the open step works fine, but trying to do a read operation results in a segfault. Furthermore, When I try to write with either:
CV_FOURCC(0,0,0,0)
0
Opencv defaults to I420 for some reason.
Next, I tried using some alternate codecs. 'DIB ' seems like something that should work for me, and on the opencv website (http://opencv.willowgarage.com/wiki/VideoCodecs) it is listed as a 'recommended' codec. However, trying to use this results in the following message:
OpenCV-2.4.1/modules/highgui/src/cap_gstreamer.cpp:483: error: (-210) Gstreamer Opencv backend doesn't support this codec acutally. in function CvVideoWriter_GStreamer::open
Aborted (core dumped)
I checked the opencv source for this codec, and stumbled across the following:
cd OpenCV-2.4.1/modules
grep -i -r "CV_FOURCC" ./*
...
./highgui/src/cap_qt.cpp: /*if( fourcc == CV_FOURCC( 'D', 'I', 'B', ' ' ))
./highgui/include/opencv2/highgui/highgui_c.h:#define CV_FOURCC_DEFAULT CV_FOURCC('I', 'Y', 'U', 'V') /* Use default codec for specified filename (Linux only) */
I tried installing qt4 and reconfiguring with the WITH_QT flag, but that did not change anything. I also tried uncommenting that part of the code and reinstalling opencv, but that also did not work.
My ultimate goal is for any way to efficiently store and retrieve a video stream with 16 bits for every pixel (like 32float would work fine, and then it wouldn't need to be perfect). Right now I am unpacking the 16 bits into the red and green channels, which is why I need it to be perfect - since an error of 1 in the red channel is multiplied by 256 in the final result. I am not having success with any of the fourcc codes available to me.
Most probably you uninstalled or updated a codec. Try install a new codec pack, or update your ffmpeg, or gstreamer
I ended up figuring this out a little while ago, and finally got a chance to write it up for everyone. You can see my (rather hacky) solution here:
http://denislantsman.com/?p=111
Edit: As the website is down the following summarizes what can be found from the Wayback Machine:
Save frames as individual PNG images
Run ffmpeg to generate a file which can be opened by OpenCV:
ffmpeg -i ./outimg/depth%d.png -vcodec png depth.mov
The following Python snippet may be useful for saving the individual frames
std::ostringstream out_depth;
...
expand_depth(playback.pDepthMap, expanded_depth, playback.rows, playback.cols);
out_depth << root << "/outimg/depth" << framecount << ".png";
cv::imwrite(out_depth.str(), expanded_depth);
framecount++;
...
Why not use FFV1? Its compression rate is way better then dibs and it is widely available.
VideoWriter video("lossless.mkv", VideoWriter::fourcc('F','F','V','1'),FPS, Size(WIDTH,HEIGHT));

Resources