I am using an implementation of YOLOv4 to detect traffic from a stream.
However because the hardware is a little bit slower than the 25 FPS of the stream, it stays behind it about 2 seconds. This means that i get two seconds jumps repeatedly throughout the detection process.
Is there a way for the OpenCV to read the stream sequentially, even when it is not synced?
Thanks
Related
I am looking to capture video on an iPhone and to initiate capture once fast motion is identified and to stop when there is slow motion or no motion is detected.
Here is a use case to illustrate:
If someone is holding the iPhone camera and there is no background movement, but his hands are not steady and moving left/right/up/down slowly, this movement should be considered slow.
If someone runs into the camera field of view quickly, this would be considered fast movement for recording.
If someone slowly walks into the camera field of view, this would be considered slow and shouldn't be picked up.
I was considering OpenCV and thought it maybe overkill using their motion detection and optical flow algorithms. I am thinking of a lightweight method by accessing the image pixel directly, perhaps examining changes in luminosity/brightness levels.
I only need to process 30-40% of the video frame area for motion (e.g. top half of screen), and can perhaps pick up every other pixel to process. The reason for a lightweight algorithm is because it will need to be very fast < 4ms as it will be processing incoming video buffer frames at a high frame rate.
Appreciate any thoughts into alternative image processing / fast motion detection routines by examining image pixels directly.
dense optical flow like calcOpticalFlowFarneback
using motion history
2.1 updateMotionHistory(silh, mhi, timestamp, MHI_DURATION);
2.2 calcMotionGradient(mhi, mask, orient, MAX_TIME_DELTA, MIN_TIME_DELTA...
2.3 segmentMotion(mhi, segmask, regions, timestamp, MAX_TIME_DELTA);
2.4 calcGlobalOrientation(orient_roi, mask_roi, mhi_roi, ...
I am working on an app that will allow users to record from the mic, and I am using audio units for the purpose. I have the audio backend figured out and working, and I am starting to work on the views/controls etc.
There are two things I am yet to implement:
1) I will be using OpenGL ES to draw waveform of the audio input, there seems to be no easier way to do it for real-time drawing. I will be drawing inside a GLKView. After something is recorded, the user should be able to scroll back and forth and see the waveform without glitches. I know it's doable, but having a hard time understanding how that can be implemented. Suppose, the user is scrolling, would I need to re-read the recorded audio every time and re-draw everything? I obviously don't want to store the whole recording in memory, and reading from disk is slow.
2) For the scrolling etc., the user should see a timeline, and if I have an idea of the 1 question, I don't know how to implement the timeline.
All the functionality I'm describing is do-able since it can be seen in the Voice Memos app. Any help is always appreciated.
I have done just this. The way I did it was to create a data structure that holds different "zoom levels" data for the audio. Unless you are displaying the audio at a resolution that will display 1 sample per 1 pixel, you don't need every sample to be read from disk, so what you do is downsample your samples to a much smaller array that can be stored in memory ahead of time. A naive example is if your waveform were to display audio at a ratio of 64 samples per pixel. Lets say you have an array of 65536 stereo samples, you would average each L/R pair of samples into a positive mono value, then average 64 of the positive mono values into one float. Then your array of 65536 audio samples can be visualized with an array of 512 "visual samples". My real world implementation became much more complicated than this as I have ways to display all zoom levels and live resampling and such, but this is the basic idea. It's essentially a Mip map for audio.
In reading about the PowerVR alpha drivers for Vulkan, they note that multibufering needs to be performed explicitly. Since Vulkan and Metal are so similar, can I actually turn off multibuffering altogether? I am willing to sacrifice throughput for low latency.
http://blog.imgtec.com/powervr/trying-out-the-new-vulkan-graphics-api-on-powervr-gpus
As a bonus, is it possible to avoid double buffering? I know racing the beam is coming back into style on the desktop but I don't know if mobile display tech supports simple single-buffering.
Double buffering is not about throughput and in fact for most modern GPUs the latency increases in single buffered operation.
I know racing the beam is coming back into style on the desktop
Most certainly not, because racing the beam works only if you can build your image scanline by scanline. However realtime graphics these days operate by sending triangles to the GPU to rasterize and the GPU has some leeway in the order in which it is touching the pixels. Also the order of the triangles relative to the screen continuously changes.
Last but not least all modern graphics systems these days are composited, which goes contrary to racing the beam.
I have a issue in programming for the graphics Card.
I write a shader for Image processing (I use DirectX 11 in combination with SharpDX), but the Transfer from the CPU to the GPU is really slow (sometime about 0.5 seconds), but I think there should be a faster way than using Texture2D.FromStream, because when playing a video, also every image of the Video has to be transfered to the GPU (or am I wrong at this Point?).
How can I Speed up this process?
does anyone know of an easy way to add a constant latency (about 30 ms) to the graphical output of an XNA 4 application?
I want to keep my graphical output in sync with a real-time buffered audio stream which inherently has a constant latency.
Thanks for any ideas on this!
Max
If you really need to delay your graphics, then what you could do is render your game to a cycling series of render-targets. So on frame n you display the frame you rendered at frame n-2. This will only work for small latencies, and requires a large amount of additional graphics memory and a small amount of extra GPU time.
A far better method is not to delay the graphical output at all, but delay the audio that is being used to generate the graphical output. Either by buffering it or having two read positions in your audio buffer. The "audio" read being X ms (the latency) ahead of the "game" read.
So if your computer's audio hardware has 100ms of latency (not uncommon), and your graphics hardware has a latency of 16ms: As you are feeding the sample at 100ms into the audio system, you are feeding the audio sample at 16ms into the your graphics calculation. At the same time, the audio from 0ms is hitting the speakers, and the matching graphic is hitting the screen.
Obviously this won't work if the thing generating the graphical output is also generating the audio. But the general principal of both these methods is that you have to buffer the input somewhere along your graphics chain, in order to introduce a delay that corresponds to the one you are experiencing for audio. Where along that chain it is easiest to insert a buffer is up to you.
For latencies of <100ms, I wouldn't worry about it for most games. You only really care about this kind of latency for audio programs and rhythm games.
I might not understand the question, but couldn't you keep track of how many times update is called and mod 2? 60fps mod 2 is 30...