I am working on my final project in Software Engineering B.Sc. Our project includes tracking a ball in Foosball game. Actually with the size of the Foosball table I will need at least HD 1080p format (1920x1080 pixels) camera and because of the high speed I will also need 60 fps.
I will use OpenCV opensource to write code in C/C++ and detect a ball on each received frame.
So here is my issue: I need to get steam from the HD camera with 60fps, Wide-angled.
I can't use a web-cam because it will not give me HD format with 60fps
(webcams can't do this, even expensive Logitech or Microsoft while it is written on the package - actually they mean that it can be low resolution with 60 fps OR HD with 30 fps) Also it is not wide-angled.
On the other hand I would like to use a web camera because it is easy to get stream out of it.
The preferred solution is to use extreme camera (something like Go Pro but cheaper version - I have AEE S70 - about 120$) I can use HDMI output of this camera to stream data to PC. But I can't use USB, it will be recognized as a Mass Storage Device. It has micro HDMI output but I have no HDMI Input on my PC.
The question is if it is possible to find some cheap capture device (HDMI->USB3.0/PCI Express) which can stream frames as HD 1080p and 60fps from this extreme camera to PC via HDMI? What device should I use? Maybe you suggest me another camera/or better solution?
Thanks
I've been looking into this for a sport application (Kinovea). It is virtually impossible to find 1080p # 60fps due to the limits of USB 2.0 bandwidth. Actually even for lower bandwidth the camera needs to perform compression on-board.
The closest camera I found is the ELP-USBFHD01M, it's from a Chinese manufacturer and can do 720p # 60fps on the MJPEG stream. I've written a full review in the following blog post.
The nice thing about this camera for computer vision is that it has a removable M12 lens, so you can use a wide angle if you want. They sell various versions of the board with pre-mounted lenses of 140°, 180°, etc.
MJPEG format means that you'll have to decompress on the fly if you want to process each image though.
Other solutions we have explored were USB 3.0 cameras but as you mention they aren't cheap and for me the fact that they don't do on-board compression was a drawback for fast recording to disk.
Another option I haven't had time to fully investigate is the HD capture cards for gamers like AVerMedia. These cards supposedly capture HD at high speed and can stream it to central memory.
Do you really need real-time processing? If you could perform the tracking on video files that you have recorded by other means you could use even 120fps files from the GoPro and get even better results.
Your choice of 1080p with 60 fps is good for the tracking application is good and as you said most of the web cams don't support such high resolution / frame rate combinations. Instead of going for a HDMI->USB3.0/PCI Express converter for your AEE S70 (which will increase the camera latency, cost and time for you to find a solution), you can check See3CAM_CU30 which streams 1080P60 uncompressed data over USB 3.0 off the shelf. Also it costs similar to your AEE S70.
Related
Im using raspberry Pi 4 B and have installed ROS melodic. I have Raspberry Pi Camera V2.1. Would like to send a compressed video with a low latency as much as possible to the Microrcontroller (ESP32) via sonar. So its important to have a very low latency as the sonar has a low bandwidth. I look at this github camera node raspberry pi camera node for Pi Camera V2 but the compressed video has a latency of more than 2 seconds. Any other way or other approaches or other help to overcome the issue with the latency?
Thanks
High latency won't necessarily affect a system with low bandwidth so long as the transmission is consistent. With any camera being processed through ROS there will almost always be some delay. The above node will probably be one of your best bets, however, there is the usb cam node. If neither of these are sufficient you'll probably need to sit down and crunch the numbers to make sure you actually have enough bandwidth/processing power. Then you might want to look into creating your own video streaming node that's a little more tailor made and lower overhead; I'd suggest gstreamer for this.
I'm looking mediaconvert service from aws to transcode videos. The value I'm trying to set just now is quality level (QL) for QVBR, according with this it could depends on the platform, for example for 720p/1080p resolution it proposes QL=8/9 (for TV), QL=7 (for tablet), QL=6 (for smartphone).
In fact, the app have a version for the 3 type of devices then I'm asking: I need to keep 3 versions for the same video? I want to save some money in streaming and my app has similar number of users using it in each platform, I want to save in bandwidth but providing good quality videos
Higher QVBR quality levels (QL) correspond to higher bitrates in the output.
For a large display such as a TV, a higher QVBR QL is recommended to help improve the viewer experience. But when viewing the same content on a smaller display such as a phone you may not need all of those extra bits to still have a good experience.
In general, it's recommended to create an output targeted for each of the various devices or resolutions content will be viewed on. This will help save bandwidth for the smaller devices while still delivering high quality for the larger ones.
This concept is referred to as Adaptive Bitrate (ABR) Streaming, and is a common feature of streaming formats such as HLS and DASH (among others). The MediaConvert documentation has a section on how to create ABR outputs as well: https://docs.aws.amazon.com/mediaconvert/latest/ug/video-abr-streaming-outputs.html
I saw that a few video players (e.g. AVPlayerHD) are doing hardware-accelerated playing on iOS for unsupported containers like MKV. How do you think they're achieving that?
I'm thinking reading packet with ffmpeg, decoding with Core Video. Does that make sense? I'm trying to achieve the same.
Thanks!
I think that the HW accelerators for video rendering (decoding) support fixed formats, due to hard wired logic. I don't know of any HW accelerator to be able to transcode from MKV.
Another method of accelerating video playback, would be the usage of OpenCL and make use of the integrated GPU on your device. This method enables HW acceleration of a wider area of applications.
The problem with this approach is that if you are not lucky enough to find a framework that uses OpenCL to do GPU acceleration of transcode / decode, you probably need to do it yourself.
Added info
To implement a fully HW accelerated solution you first need to transcode the MKV into H264 & sub, and from there you can use the HW decoder to render the H264 component.
For the HW accelerated transcode operations you could use GPU (via OpenCL) and/or multithreading.
I found https://handbrake.fr/ that might have some OpenCL transcoding features.
Cheers!
I've been researching for a while and found tons of helpful resources on this subject, but I figured I would lay down my specifications here so I can get some recommendations from people experienced in this area. It seems like Beaglebone and Raspberry Pi with a Logitech or Microsoft camera are my best options at this point.
My target speed is 50 fps (20 ms per image) with the processing involved. From what I've looked at, this doesn't seem feasible considering most webcams don't go much past 30 fps. More specifically, I need to take the endpoints of an object (like a sheet of paper) and calculate where the midpoint is. Nothing incredibly fancy. 1080p isn't a requirement, I can most likely go much lower. Python is preferable over C and C++ since I've already done a lot of image processing with Python.
It looks like a lot of the code I'll be needing is mostly open-source already, so I really just need to figure out what controller/camera combo I should be using.
It's still a bit of a toss up between the two however here are my views.
The BBB will use a USB web cam and that will take a certain amount of processing power just to get the image. After that you can then manipulate it with SimpleCV
The RPi has a camera board that they say will only use < 3% of the cpu and the rest can be used for processing your image. Plus you can over-clock the RPi to 1Ghz.
Using the RPi with a basic webcam does not give a very good result, whereas the RPi camera works directly on the CSI bus and is set to do 1080 dpi natively. Plus they now have drivers for the camera that work with SimpleCV too.
IMHO I would say that the RPi B and Camera board would be technically faster that the BBB, but it also depends on what manipulation you plan to do :
Marc
I am currently in a webcam streaming server project that requires the function of dynamically adjusting the stream's bitrate according to the client's settings (screen sizes, processing power...) or the network bandwidth. The encoder is ffmpeg, since it's free and open sourced, and the codec is MPEG-4 part 2. We use live555 for the server part.
How can I encode MBR MPEG-4 videos using ffmpeg to achieve this?
The multi-bitrate video you are describing is called "Scalable Video Codec". See this wiki link for basic understanding.
Basically, in a scalable video codec, a base layer stream itself has completely decodable; however, additional information is represented in the form of (one or many) enhancement streams. There are couple of techniques to be able to do this including lower/higher resolution, framerate and change in Quantization. The following papers explains in details
of Scalable Video coding for MEPG4 and H.264 respectively. Here is another good paper that explains what you intend to do.
Unfortunately, this is broadly a research topic and till date no open source (ffmpeg and xvid) doesn't support such multi layer encoding. I guess even commercial encoders don't support this as well. This is significantly complex. Probably you can check out if Reference encoder for H.264 supports it.
The alternative (but CPU expensive) way could be transcode in real-time while transmitting the packets. In this case, you should start off with reasonably good quality to start with. If you are using FFMPEG as API, it should not be a problem. Generally multiple resolution could still be a messy but you can keep changing target encoding rate.