Mixing video and data streams on raspberry pi for machine learning - machine-learning

What is best way to mix data and video for machine learning on single board computer like raspberry pi?
It should be very common problem. I found there is GPMF and KLV extensions for video formats. But I fail to find someone to implment that for raspberry pi, opencv, etc.
It seems I can solve the problem by mixing 2 data streams in single file and then read that file with my own code but I wonder what is canonical solution for that kind of problem.

Related

Feeding multiple video streams to one ML model using multiprocessing in Python

Currently I have a system of ML models that run in their own processes in Python. It works perfectly when a single video camera feed is input but now I need to feed video from multiple sources and only have resources to run one instance of the model. I tried to batch process multiple video streams but it is not really scalable for more than 5 cameras. Is there any Python framework or pipeline that could be helpful? Please suggest.

sfm/slam implementing with webcam

I want to implement Structur from Motion (SfM) / Simultaneous Localization and mapping algorithms using my webcam. I am very new on this topic so I need advices from experts in the internet. I could now able to build OpenCV opencv sfm tutorial for this purpose and I looked OpenSFM but it seems like just a GUI. What other open libraries/programs that I can use for this task? any suggestions/advices/tutorials are appreciated.

Using OpenCV to create a Neural network?

I am working on creating a Real-time image processor for a self driving small scale car project for uni, It uses a raspberry pi to get various information to send to the program to base a decision by.
the only stage i have left is to create a Neural network which will view the image displayed from the camera ( i already have to code to send the array of CV_32F values between 0-255 etc.
I have been scouring the internet and cannot seem to find any example code that is related to my specific issue or my kind of task in general (how to implement a neural network of this kind), so my question is is it possible to create a NN of this size in c++ without hard coding it (aka utilising openCv's capabilities): it will need 400 input nodes for each value (from 20x20 image) and produce 4 outputs of left right fwd or backwards respectively.
How would one create a neural network in opencv?
Does openCV provide a backpropogation(training) interface /function or would I have to write this myself.
once it is trained am I correct in assuming I can load the neural network using ANN_MLP load etc? following this pass the live stream frame (as an array of values) to it and it should be able to produce the correct output.
edit:: I have found this OpenCV image recognition - setting up ANN MLP. and It is very simple in comparison to what I want to do, and I am not Sure how to adapt that to my problem.
OpenCV is not a neural network framework and in turn won't find any advanced features. It's far more common to use a dedicated ANN library and combine it with OpenCV. Caffe is a great choice as a computer vision dedicated deep learning framework (with C++ API), and it can be combined with OpenCV.

openCV compatible webcams

I have read that openCV does not recognise some webcams.
Can you suggest some webcams for a basic stereo-vision project using the following?:
openCV 2.2.0
Visual Studio 2010 (C++)
Windows 7 32 bits
Thanks.
I posted a similar question back then.
Good and compatible webcam to do image processing/computer vision?
What kind of camera you are looking for? Any model? State the model and I let you know if it's good enough.
Also, you can find a list of compatible webcams with openCV. It's an old list though. They stopped updating it since 2013, I think. Compatible webcam with openCV
You find that most common brands like Hp, Logitech, etc should work just fine. If you are worried, just state the model you are looking at in the comments. I see if I know if it's compatible with openCV.
EDIT:
To answer your question in the comment, nope, that shouldn't be the case. All cameras can run simultaneously at the same time. Reasons that cameras can't run simultaneously at the same time is usually due to the USB bandwidth. In fact, most cameras pose the same problem.
Some methods to overcome that problem:
1)put a Sleep(ms) in between the lines of your capture line.
2)Use lower resolution which would reduce the bandwidth used by each camera.
3)Use MJPEG format(compressed frames)

OpenCV + Webcam compatibility

For the people that have experience with OpenCV, are there any webcams that don't work with OpenCV.
I am looking into the feasibility of a project and I know I am going to need a high quality feed (1080p), so I am going to need a webcam that is capable of that. So does OpenCV have problems with certain cameras?
To be analysing a video feed of that resolution on the fly I am going to need a fast processor, I know this, but will I need a machine that is not consumer available...ie, will an i7 do?
Thanks.
On Linux, if it's supported by v4l2, it is probably going to work (e.g., my home webcam isn't listed, but it's v4l2 compatible and works out of the box). You can always use the camera manufacturer's driver to acquire frames, and feed them to your OpenCV code. You can even sub-class the VideoCapture class, and implement your camera driver to make it work seamlessly with OpenCV.
I would think the latest i7 series should work just fine. You may want to also check out Intel's IPP library for more optimized routines. IPP also easily integrates into OpenCV code since OpenCV was an Intel project at its inception.
If you need really fast image processing, you might want to consider adding a high performance GPU to the box, so that you have that option available to you.
Unfortunately, the page that I'm about to reference doesn't exist anymore. OpenCV evolved a lot since I first wrote this answer in 2011 and it's difficult for them to keep track of which cameras in the market are supported by OpenCV.
Anyway, here is the old list of supported cameras organized by Operating System (this list was available until the beginning of 2013).
It depends if your camera is supported by OpenCV, mainly by the driver model that your camera is using.
Quote from Getting Started with OpenCV capturing,
Currently two camera interfaces can be used on Windows: Video for Windows (VFW) and Matrox Imaging Library (MIL) and two on Linux: Video for Linux(V4L) and IEEE1394. For the latter there exists two implemented interfaces (CvCaptureCAM_DC1394_CPP and CvCapture_DC1394V2).
So if your camera is VFW or MIL compliant under Windows or suits into standard V4L or IEEE1394 driver model, then probably it will work.
But if not, like mevatron says, you can even sub-class the VideoCapture class, and implement your camera driver to make it work seamlessly with OpenCV.

Resources