I am working on some Machine Learning project where I have tracked people with Kalman Filter tracker. I want to calculate that how much time each person is there in the video.
I tried using the following logic :
Suppose the person is present in the 5 frames and the video FPS is 15, then we can say that the person is (5*15) for 75 seconds in the video.
Note:- I have assumed & hard coded the FPS value in the code. I didn't find any way to get the FPS because I am passing frames of video for tracking.
But the problem is if I hard code the FPS value, so whenever the FPS changes (which I don't know when), I have to change in code, otherwise it will give wrong result.
You don't need to hard code fps and you infact dont need fps for this. I believe your tracker will assign unique id to each person detected. Once you have the id, you can sinply start counting the number of seconds for that id. Once that person moves out of the frame, tracker id will be lost and you can stop the timer and thus you will have the total time person spent in the frame.
Have a look at this code: https://github.com/mailrocketsystems/AIComputerVision/blob/master/dwell_time_calculation.py
and maybe this video for explanation: https://www.youtube.com/watch?v=qn26XSinYfg
My suggestion is to calculated elapsed time for processing each captured frame and maintaining an accumulator to calculate the on-screen time for each person detected. The frame rate depends on the amount of processing you do, for this type of projects.
Related
I want to know if setting a delay in seconds to the camera feed is possible on iOS with Swift.
Let's say I add a 30 second delay to my camera. If I am pointing the camera at a person who is crouching down, and then that person stands up, my camera is not going to show that until those 30 seconds have passed. In other words, it will continue to show the person crouched until that time has passed.
how can I achieve this?
This sounds extremely difficult. Problem is camera recording is bringing in (most likely) 30 frames a second. So those buffers from 30 seconds ago need to be saved somewhere, and in raw form that is a TON of data so you can't just keep those in memory.
You don't want to write as images because the compression is garbage compared to video.
Perhaps the best way would be to setup your capture, then capture your video in say 10 second segments, save to file, then display those video file segments on an AVPlayer or something. That way it's 'delayed' for the user.
Either way, from my understanding of how things work, the main challenge is what to do with those buffers while waiting to display them. You could potentially stream it somewhere, but then you need a whole backend to support that and then stream it back, seems silly.
I have a problem with the iOS SDK. I can't find the API to slowdown a video with continuous values.
I have made an app with a slider and an AVPlayer, and I would like to change the speed of the video, from 50% to 150%, according to the slider value.
As for now, I just succeeded to change the speed of the video, but only with discrete values, and by recompiling the video. (In order to do that, I used AVMutableComposition APIs.
Do you know if it is possible to change continuously the speed, and without recompiling?
Thank you very much!
Jery
The AVPlayer's rate property allows playback speed changes if the associated AVPlayerItem is capable of it (responds YES to canPlaySlowForward or canPlayFastForward). The rate is 1.0 for normal playback, 0 for stopped, and can be set to other values but will probably round to the nearest discrete value it is capable of, such as 2:1, 3:2, 5:4 for faster speeds, and 1:2, 2:3 and 4:5 for slower speeds.
With the older MPMoviePlayerController, and its similar currentPlaybackRate property, I found that it would take any setting and report it back, but would still round it to one of the discrete values above. For example, set it to 1.05 and you would get normal speed (1:1) even though currentPlaybackRate would say 1.05 if you read it. Set it to 1.2 and it would play at 1.25X (5:4). And it was limited to 2:1 (double speed), beyond which it would hang or jump.
For some reason, the iOS API Reference doesn't mention these discrete speeds. They were found by experimentation. They make some sense. Since the hardware displays video frames at a fixed rate (e.g.- 30 or 60 frames per second), some multiples are easier than others. Half speed can be achieved by showing each frame twice, and double speed by dropping every other frame. Dropping 1 out of every 3 frames gives you 150% (3:2) speed. But to do 105% is harder, dropping 1 out of every 21 frames. Especially if this is done in hardware, you can see why they might have limited it to only certain multiples.
Im programming an iOS game and I use the method update for a lot of things, which is called at the game speed refresh (for the moment 60 times per second) but the problem is if the frame rate drops down (for example a notification, or any behavior in the game that, when called, it makes drop down a little bit the fps...) then the bugs comes....
A fast example is if I have an animation of 80 pictures, 40 for jump up and 40 for fall, I would need 1,2 second to run the animation, so if the jump takes 1,2 second it would be ok, the animation would run. But if my fps drop down to 30 then the animation would cut because it would need 2,4 seconds to run the animation but the jump remains 1,2 second. This is only a fast example, there is a lot of unexpected behaviors in the game if the frame rate drops, so my question is, are games developers depending so much on frame rate or there is a way to avoid those fps-bugs? (another way to program or any trick?)
Base your timing on the time, rather than the frame count. So, save a time stamp on each frame, and on the next frame, calculate how much time has elapsed, and based on your ideal frame rate, figure out how many frames of animation to advance. At full speed, you shouldn’t notice a difference, and when the frame rate drops, your animations may get jerky but the motion will never get more than 1 frame behind where it should be.
Edit: as uliwitness points out, be careful what time function you use, so you don’t encounter issues when, for example, the computer goes to sleep or the game pauses.
Always use the delta value in your update method. This is platform and engine independent. Multiply any speed or change value by the delta value (the time interval between the current and the last frames).
In case of the animation, one way to fix the issue could be to multiply the animation counter by delta (and an inverse of the expected interval). Then round this value to get the correct image for the animation.
// currentFrame is a float ivar set to 0 at the beginning of the animation.
currentFrame = currentFrame + delta * 60.0;
int imageIndex = roundf(currentFrame);
However, with Sprite Kit there is a better way to do this kind of animation, as there is a premade SKAction dealing with sprite animation.
[SKAction animateWithTextures:theTextures timePerFrame:someInterval];
With this solution you don't have to deal with timing the images at all. The engine will do that for you.
There's a great discussion about FPS-based and Time-based techniques here:
Why You Should be Using Time-based Animation and How to Implement it
It's the best on my opinion, very complete, easy to follow and provides JsFiddle examples. I translated those examples to C++/Qt.
Click here to watch a video of my app running:
I am attempting to record an animation (computer graphics, not video) to a WMV file using DirectShow. The setup is:
A Push Source that uses an in-memory bitmap holding the animation frame. Each time FillBuffer() is called, the bitmap's data is copied over into the sample, and the sample is timestamped with a start time (frame number * frame length) and duration (frame length). The frame rate is set to 10 frames per second in the filter.
An ASF Writer filter. I have a custom profile file that sets the video to 10 frames per second. Its a video-only filter, so there's no audio.
The pins connect, and when the graph is run, a wmv file is created. But...
The problem is it appears DirectShow is pushing data from the Push Source at a rate greater than 10 FPS. So the resultant wmv, while playable and containing the correct animation (as well as reporting the correct FPS), plays the animation back several times too slowly because too many frames were added to the video during recording. That is, a 10 second video at 10 FPS should only have 100 frames, but about 500 are being stuffed into the video, resulting in the video being 50 seconds long.
My initial attempt at a solution was just to slow down the FillBuffer() call by adding a sleep() for 1/10th second. And that indeed does more or less work. But it seems hackish, and I question whether that would work well at higher FPS.
So I'm wondering if there's a better way to do this. Actually, I'm assuming there's a better way and I'm just missing it. Or do I just need to smarten up the manner in which FillBuffer() in the Push Source is delayed and use a better timing mechanism?
Any suggestions would be greatly appreciated!
I do this with threads. The main thread is adding bitmaps to a list and the recorder thread takes bitmaps from that list.
Main thread
Animate your graphics at time T and render bitmap
Add bitmap to renderlist. If list is full (say more than 8 frames) wait. This is so you won't use too much memory.
Advance T with deltatime corresponding to desired framerate
Render thread
When a frame is requested, pick and remove a bitmap from the renderlist. If list is empty wait.
You need a threadsafe structure such as TThreadList to hold the bitmaps. It's a bit tricky to get right but your current approach is guaranteed to give to timing problems.
I am doing just the right thing for my recorder application (www.videophill.com) for purposes of testing the whole thing.
I am using Sleep() method to delay the frames, but am taking great care to ensure that timestamps of the frames are correct. Also, when Sleep()ing from frame to frame, please try to use 'absolute' time differences, because Sleep(100) will sleep about 100ms, not exactly 100ms.
If it won't work for you, you can always go for IReferenceClock, but I think that's overkill here.
So:
DateTime start=DateTime.Now;
int frameCounter=0;
while (wePush)
{
FillDataBuffer(...);
frameCounter++;
DateTime nextFrameTime=start.AddMilliseconds(frameCounter*100);
int delay=(nextFrameTime-DateTime.Now).TotalMilliseconds;
Sleep(delay);
}
EDIT:
Keep in mind: IWMWritter is time insensitive as long as you feed it with SAMPLES that are properly time-stamped.
in image processing applications what is considered real time? Is 33 fps real time? Is 20 fps real time? If 33 and 20 fps are considered real time then is 1 or 2 fps also real time?
Can anyone throw some light.
In my experience, it's a pretty vague term. Often, what is meant is that the algorithm will run at the rate of the source (e.g. a camera) supplying the images; however, I would prefer to state this explicitly ("the algorithm can process images at the frame rate of the camera").
Real time image processing = produce output simultaneously with the input.
The input may be 25 fps but you may choose to process 1 of every 5 frames(that makes 5 fps processing) and your application is still real time.
TV streaming software: all the frames are processed.
Security application and the input is CCTV security cams: you may choose to skip some frames to fit the performance.
3d game or simulation: fps changes depending on the current scene.
And they are all real time.
Strictly speaking, I would say real-time means that the application is generating images based on user input as it occurs, e.g. a mouse movement which changes the facing of an avatar.
How successful it is at this task - 1 fps, 10 fps, 100 fps, etc - is actually another question.
Real-time describes an approach, not a performance metric.
If however you ask what is the slowest fps which passes as usable by a human, the answer is about 15, I think.
i think it depends on what the real time application is. If the app is showing slideshows with 1 picture every 3 seconds, and the app can process 1 picture within this 3 seconds and show it, then it is real time processing.
If the movie is 29.97 frames per second, and the app can process all 29.97 frames within the second, then it is also real time.
An example is, if an app can take the movie from a VCR or Cable's analog output, and compress it into 29.97 frames per second video and also send all that info to a remote area for another person to watch, then it is real time processing.
(Hard) Real time is when an outcome has no value when delivered too early or too late.
Any FPS is real time provided that displayed frames represent what should be displayed at the very instant they are displayed.
The notion of real-time display is not really tied to a specific frame rate - it could be defined as the minimum frame rate at which movement is perceived as being continuous. So for slow moving objects in a visual frame (e.g. ships in a harbour, or stars in the night sky) a relatively slow frame rate might suffice, whereas for rapid movement (e.g. a racing car simulator) a much higher frame rate would be needed.
There is also a secondary consideration of latency. A real-time display must have sufficiently low latency in relation to other events (e.g. behaviour of a real-time simulation) that there is no perceptible lag in display updates.
That's not actually an easy question (even without taking into account differences between individulas).
Wikipedia has a good article explaining why. For what it's worth, I think cinema films run at 24fps so, if you're happy with that, that's what I'd consider realtime.
It depends on what exactly you are trying to do. For some purposes 1fps or even 2 spf (Seconds per frame) could be considered real-time. For others thats way too slow ...
That said, real-time means that it takes as long (or less) to process x frames as it would take to just present those x frames.
It depends.
automatic aircraft cannon - 1000 fps
monitoring - 10 - 15 fps
authentication - 1 fps
medical devices - 1 fph
I guess the term is used with different meanings in different contexts. In industrial image processing, real time processing is usually the opposite of offline processing. In offline processing applications, you record images (many of them) and process them at a later time. In real time processing, the system that acquires the images also processes them, at the same time, so the processing frame rate must not be higher than the acquisition frame rate.
Real-time means your implementation is fast enough to meet some deadline. The deadline is part of your system's specification. If it's an interactive UI and the users are not too picky, 15Hz update can be OK, although it can feel laggy. If you're using it to drive a car along the motorway 30Hz is about right. If it's a missile, well, maybe 100Hz?