How to deal with cv::VideoCapture decode errors? - opencv

I'm streaming H264 content from an IP camera using the VideoCapture from OpenCV (compiled with ffmpeg support).
So far things work ok, but every once in a while I get decoding errors (from ffmpeg I presume):
[h264 # 0x103006400] mb_type 137 in I slice too large at 26 10
[h264 # 0x103006400] error while decoding MB 26 10
[h264 # 0x103006400] negative number of zero coeffs at 25 5
[h264 # 0x103006400] error while decoding MB 25 5
[h264 # 0x103006400] cbp too large (421) at 35 13
[h264 # 0x103006400] error while decoding MB 35 13
[h264 # 0x103006400] mb_type 121 in P slice too large at 20 3
[h264 # 0x103006400] error decoding MB 20 3
These messages show up in the console. Is there any clean way to listen to these ? I'd like to skip processing the glitchy frames.
Any hints/tips ?

recently i have solved the same problem and try to explain the steps i followed.
i updated most recent opencv_ffmpeg.dll ( i renamed opencv_ffmpeg.dll to opencv_ffmpeg310.dll to use with OpenCV 3.1, also renamed same dll opencv_ffmpeg2412.dll to use with OpenCV 2.4.12
by doing that, a basic capturing frames and display became successful without problem.but still the same problem if i do some image-processing or detection causes delay between capturing frames.
to solve the second problem i used a thread to grab frames continiously and update a global Mat for processing.
here
you can find my test code ( it need some improvements like using mutex and lock memory when update the Mat)
i hope the information will be useful ( sorry for my poor english )

I have the same problem. It seems to me that the problem comes from the fact that the source originating the stream is slower than the one decoding. Probably for the decoding you have an endless loop reading frames and decoding them, which might be faster than what your source can send you.
I don't know how to stop and wait until the buffer is full .. I'm using a file, such that my camera source writes a file and I read frames from it in my decoding program. So far I haven't been able to synch them

what sturkmen said is absolutely right, the opencv version is 2413, and for some reason, I can not update the opencv to 310, I know there is gonna be no any decoding error like this for opencv310. So firstly, I copy lib opencv_ffmpeg310_64.dll to my executable file path E:\GITHUB\JpegRtspCamera\vs2013\JpegRtspCamera\x64\Release
then I just delete opencv_ffmpeg2413.dll and change the name opencv_ffmpeg310_64.dll to opencv_ffmpeg2413.dll. it works!!!

Related

ImageMagick memory consumption becomes 1.4 GB when trying to open an 876 kb animated gif

I'm trying to read an animated gif with ImageMagick. The file in question is available online, located here.
My code (linked with ImageMagick/MagickWand 7) is
#include <stdlib.h>
#include <MagickWand/MagickWand.h>
int main(void){
MagickWand *magick_wand;
MagickWandGenesis();
magick_wand = NewMagickWand();
MagickReadImage(magick_wand, "animated.gif");
return 0;
}
If I run this in the debugger and move to the line right after the image is read, the process is taking up 1.4GB of memory, according to top. I've found animated gifs with similar file sizes, and they don't go anywhere near this amount of memory consumption. Unfortunately, my experience with animated gif processing is very limited, so I'm not sure what's reasonable or not.
I have a few questions: Is this reasonable? Is it a bug? Does anyone know what makes the memory consumption of one file different from another? Is there way to control the memory consumption of ImageMagick? There's apparently a file called policy.xml which can be used to specify upper memory limits, but I've set it low and still get this behavior.
If you're curious about the larger context behind this question, in real life I'm using a python library called Wand to do this in a CMS web application. If a user uploads this particular file, it causes the OOM killer to kill the app server process (the OOM limit on these machines is set fairly low).
[Update]:
I've been able to get memory limits in policy.xml to work, but I need to set both the "memory" and "map" values. Setting either low but not the other doesn't work. I'm still curious on the other points.
ImageMagick6 decompresses the entire image to memory on load and represents each pixel as a sixteen bit number. This needs a lot of memory! ImageMagick7 uses floats rather than 16 bit numbers, so it'll be twice the size again. Your GIF is 1920 x 1080 RGBA pixels and has 45 frames, so that's 1920 * 1080 * 45 * 4 * 4 bytes, or about 1.4gb.
To save memory, you can get IM to open large images via a temporary disk file. This will be easier on your RAM, but will be a lot slower.
Other image processing libraries can use less memory -- for example libvips can stream images on demand rather than loading them into RAM, and this can give a large saving. With your image and pyvips I see:
$ python3
Python 3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> import os, psutil
>>> process = psutil.Process(os.getpid())
>>> # n=-1 means load all frames, access="sequential" means we want to stream
>>> x = pyvips.Image.new_from_file("huge2.gif", n=-1, access="sequential")
>>> # 50mb total process size after load
>>> process.memory_info().rss
49815552
>>> # compute the average pixel value for the entire animation
>>> x.avg()
101.19390990440672
>>> process.memory_info().rss
90320896
>>> # total memory use is now 90mb
>>>

OpenCV how to grab the real time frame and not the next frame?

I realized that when I use OpenCV to grab videos (cv2.VideoCapture('rtsp://...')) in a rtsp url, actually I am getting everyframe of the stream and not the real time frame.
Example: If a video has 30fps and 10 seconds long, if I get the first frame and wait 1 second to get the next, I get the frame number 2 and not the real time frame (it should be frame number 30 or 31).
I am worried about these because if my code take a little longer to do the video processing (deep learning convolutions), the result will always be delivered later and not in real time.
Any ideas how can I manage to always get the current frame when I capture from rtsp?
Thanks!
This is not about the code. Many IP cameras gives you encoded output(H.265/H.264).
When you used VideoCapture() , the output data of the camera is decoded by the CPU. Getting delay as you mentioned also such as between 1 sec and 2 sec is normal.
What can be done to make it faster:
If you have a GPU hardware, you can decode the data via on it. This
will give you really good results(according to experiences by using
latest version of NVIDIA gpus: you will get almost 25 milisecond
delay) To achieve that on your code, you need:
CUDA installation
CUDA enabled OpenCV installation
VideoReader class of OpenCV
You can use VideoCapture() with FFMPEG flag, FFMPEG has advanced methods to decode encoded data and this will give you probably most faster output which you can get with your CPU. But this will not decrease time much.

Decoding a h264 (High) stream with OpenCV's ffmpeg on Ubuntu

I am working with a video stream from an ip camera on Ubuntu 14.04. Everything was going great with a camera that has these parameters (from FFMPEG):
Stream #0:0: Video: h264 (Main), yuv420p(progressive), 352x192, 29.97 tbr, 90k tbn, 180k tbc
But then i changed to a newer camera, which has these parameters:
Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1280x720, 25 fps, 25 tbr, 90k tbn, 50 tbc
My C++ program uses OpenCV3 to process the stream. By default OpenCV uses ffmpeg to decode and display the stream with function VideoCapture.
VideoCapture vc;
vc.open(input_stream);
while ((vc >> frame), !frame.empty()) {
*do work*
}
With the new camera stream i get errors like these (from ffmpeg):
[h264 # 0x7c6980] cabac decode of qscale diff failed at 41 38
[h264 # 0x7c6980] error while decoding MB 41 38, bytestream (3572)
[h264 # 0x7c6980] left block unavailable for requested intra mode at 0 44
[h264 # 0x7bc2c0] SEI type 25 truncated at 208
The image sometimes is glitched, sometimes completely frozen. However on vlc it plays perfectly. I installed the newest version (3.2.2) of ffmpeg player with
./configure --enable-gpl --enable-libx264
Now playing directly with ffplay (instead of launching from source code with OpenCV function VideoCapture), the stream plays better, but sometimes still displays warnings:
[NULL # 0x7f834c008c00] SEI type 25 size 896 truncated at 320=1/1
[h264 # 0x7f834c0d5d20] SEI type 25 size 896 truncated at 319=1/1
[rtsp # 0x7f834c0008c0] max delay reached. need to consume packet
[rtsp # 0x7f834c0008c0] RTP: missed 1 packets
[h264 # 0x7f834c094740] concealing 675 DC, 675 AC, 675 MV errors in P frame
Changing the camera hardware is not an option. The camera can be set to encode to h265 or mjpeg. When encoding to mjpeg it can output 5 fps, which is not enough. Decoding to a static video is not an option either, because i need to display real time results about the stream. Here is a list of API backends that can be used in function VideoCapture. Maybe i should switch to some other decoder and player?
From my research i conclude that i have these options:
Somehow get OpenCV to use libVlc instead of ffmpeg
One example of switching to vlc is here, but i don't understand it well enough to say if that is what i need. Or maybe i should be parsing the stream in code?
Use vlc to preprocess the stream, as suggested here.
This is probably slow, which again is bad for real time results.
Any suggestions and coments will be appreciated.
The errors are caused by packet loss since it uses RTP by default. You are seeing more errors now because you have switched to a higher bitrate input.
Append ?tcp to your input, eg. rtsp://*private*/media/video2?tcp in OpenCV's open function to force TCP, assuming it's supported by your hardware and/or usage scenario.
I have mostly solved this issue.
First i recompiled OpenCV with the newest version of ffmpeg manually installed on my device with the necessary settings.
I checked the stream with VLC tools > codec information to check if the stream has corrupted frames, which it didn't.
I lowered the stream resolution. This gave the biggest improvement.
The last error that i'm still left with is
[NULL # 0x7f834c008c00] SEI type 25 size 896 truncated at 320=1/1
[h264 # 0x7f834c0d5d20] SEI type 25 size 896 truncated at 319=1/1
but it doesn't visibly damage the frames. I have not resolved the stream freezing issue however, but it has something to do with my code, not the software. If i can help anyone with similar problems, feel free to ask additional info.
I have same problem after 4 days research finaly i solved the problem easy by this code:
for(;;) {
if(!vcap.read(image)) {
std::cout << "No frame" << std::endl;
cv::waitKey();
}
cv::imshow("Output Window", image);
if(cv::waitKey(1) >= 0) break;
}

Mov file has more frames than written/Possible iOS AVAsset writer usage issue

I am manually generated a .mov video file.
Here is a link to an example file: link, I wrote a few image frames, and then after a long break wrote approximately 15 image frames just to emphasise my point for debuting purposes. When I extract images from the video ffmpeg returns around 400 frames instead of the 15-20 I expected. Is this because the API i am using is inserting these image files automatically? Is it a part of the .mov file format that requires this? Or is it due to the way the library is extracting the image frames from the video? I have tried searching the internet but could not arrive at an answer.
My use case is that I am trying to write the current "sensor data" (from core motion) from core motion while writing a video. For each frame I receive from the camera, I use "AppendPixelBuffer" to write the frame to the video and then
Thanks for any help. The end result is I want a 1:1 ratio of Frames in the video to rows in the CSV file. I have confirmed I am writing the CSV file correctly using various counters etc. So my issue is cleariy the understanding of the movie format or API.
Thanks for any help.
UPDATED
It looks like your ffmpeg extractor is wrong. To extract only the timestamped frames (and not frames sampled at 24Hz) in your file, try this:
ffmpeg -i video.mov -r 1/1 image-%03d.jpeg
This gives me the 20 frames expected.
OLD ANSWER
ffprobe reports that your video has a frame rate of 2.19 frames/s and a duration of 17s, which gives 2.19 * 17 = 37 frames, which is closer to your expected 15-20 than ffmpeg's 400.
So maybe the ffmpeg extractor is at fault?
Hard to say if you don't show how you encode and decode the file.

Fixing a TS file made by the HD Home Run

I am recording from a cable stream using the hdhomerun command line tool, hdhomerun_config, to a .ts file. The way it works is that you run the command, it produces periods every second or so to let you know that the stream is being successfully recorded. So when I record, it produces only periods, which is desired. And the way to end it is by doing a Ctrl-C. However, whenever I try to convert this to an avi or a mov using FFMpeg, it gives a bunch of errors, some of which being
[mpeg2video # 0x7fbb4401a000] Invalid frame dimensions 0x0
[mpegts # 0x7fbb44819600] PES packet size mismatch
[ac3 # 0x7fbb44015c00] incomplete frame
It still creates the file, but it is bad quality and it doesn't work with OpenCV and other services. Has anyone else encountered this problem? Does anyone have any knowledge that may help with this situation? I tried to trim the ts file but most things require conversion before editing. Thank you!
Warnings/errors like that are normal at the very start of the stream as the recording started mid stream (ie mid PES packet) and ffmpeg expects PES headers (ie the start of the PES packet). Once ffmpeg finds the next PES header it will be happy (0-500ms later in play time).
Short version is that it is harmless. You could eliminate the warnings/errors but removing all TS-frames for each ES until you hit a payload unit start flag, but that is what ffmpeg is already doing itself.
If you see additional warnings/errors after the initial/start then there might be a reception of packet loss issue that needs investigation.

Resources