When using the MOG2 bacground subtractor in OpenCV I am getting results that display parts of the background in foreground segmented regions of the output. I am processing a live camera feed.
I am initialising with default paramaters and then changing the number of Gaussians to 5. For the first 100 frames when it is assumed that the scene is static, the GMM is updated with a learning rate of -1.0. After 100 frames, when movement is expected, a learning rate of 0.0 is used.
An example of the mask given with no movement is as follows:-
It can be seen that the mask is rather noisy, though that may be down to luminescence changes in the scene.
When placing my arm in front of the camera, it can be seen that the outline of my downstairs bannister is marked as background in the region that should be solidly my arm, as follows:-
Is this the expected behaviour for the MOG2 background subtractor? Is the update and then use for prediction model that I have employed sufficient to segment moving components in a static scene? Would one expect remnants of the occluded scene to appear in the segmentation mask?
Related
I've got a video stream from the camera. My goal is to detect and track position of moving object (train).
First of all I tried to use movement detection (frames difference, background subtractors) but it gave bad results.
Tried to cling to the color of the object but it's often (bad lighting, blurry) the same color as a ground (railways).
So the current approach is to divide the area of movement of object into n small regions and define difference between the stored region when there's no object and current one.
The main problem in here is that lightness is changing and when I use a stored region from a reference frame (where there's no object) the brightness of the current frame might be different and it breaks it up.
Also brightness can change while object moving.
It helps me to apply a gaussianBlur and histogramEqualization to make sensitivity to changes in brightness a bit less.
I tried to compare a structure of according regions using ssim, lbph, hog. When I test lbph and hog approaches on the manual cropped regions which is larger than real ones it looked like working but then I used them for my small regions and it stopped working.
In the moment the most efficient approach is just difference between grayscale regions with rmse using a fixed thresholds but it's not robust approach and it suffers a lot when brightness is changing.
Now I'm trying to use some high-pass operator to extract the most dominant edges like with sobel operator in the attached figure but I'm not sure how to properly compare the high-passed regions except by finding the difference.
Frame with an empty railway:
In some seconds a train is appeared and luminance was changed.
At night time luminance is also different.
So the questions are:
What approaches are there for comparison of high-passed images?
Is there any other way to determine if an area is overlapped which you could suggest me?
I am trying to find the positions of multiple monitors in 3d space. For that purpose I display charuco boards and take a short video of the whole setup. Sample frame here (cropped down from 4k to preserve privacy):
Code is taken verbatim from the OpenCV charuco tutorials:
https://docs.opencv.org/master/da/d13/tutorial_aruco_calibration.html
https://docs.opencv.org/3.4/df/d4a/tutorial_charuco_detection.html
The only change I made was that I created all six boards from separate parts of a big aruco dictionary so that every single marker is unique.
Calibration reports an error of 3-4 (which I believe is in pixels, so on a 4k video isn't too bad).
90% of time this works fine, but in some frames the detection fails spectacularly for no apparent reason as is visible in the lower right screen in the image above. Note that the actual Aruco markers (red corners, green frames, blue ids) get detected well, but the program fails to detect the pose of the charuco board (green markers, rgb axis). This happens sporadically for one or two frames, but I would prefer it not to happen at all.
Any ideas?
I'm working on Scene Boundary Detection and following work i have done
Conventional Approach: In this approach I've used SSIM and Pearson Coefficient to compare similarity between two frames
Advantages
Works well on Abrupt Cut Transitions
Fast in execution
Disadvantage
- False To detect gradual transition(fade-in, fade-out, wipe,dissolve, fast blur)
Deep SBD: In this I have implemented deepSBD. Link of the code and paper is https://github.com/abramjos/Scene-boundary-detection
Advantages
Works well in Abrupt cut and Gradual Transition.(fade-in, fade-out, wipe,dissolve)
Disadvantage
Included manual flash in training due to which wrong detection increases when intensive flash appears in between Frames for eg. Wrestling Videos or sudden sunlight in between shots.
Fast Blur motion
Long Dissolve Transition
Transnet: After that we have implemented Transnet. Link of code and paper is https://github.com/soCzech/TransNet
It replaces some of the wrong detection with correct detection of gradual transition but introduces new problems:
Overlap in between frames were wrongly detected as shot change
Zoomed transition in between frames were wrongly detected as shot change
NOTE: AT Last we combined deepSBD and TransNet so None detection is very less but False detection increased because of following
Intensive Flash
Fast Motion Blur
Zoomed Transition
Overlap in between Frames
Overall Accuracy for 2-3 hours video is expected to be 95% if above problems doesn't appear and if it appears then 70%.
Is there any way to resolve False detection .
I have a background image of a plain surface.
My goals is to track objects that are positioned/moved over the surface.
I'm using MOG2 to find foreground objects with a learning rate of 0, so the background is not updated (otherwise a static object would be incorporated in the background).
The result is fine, but I have a huge problem with light: if the lighting changing after background acquiring, various artifacts are detected as foregorund objects.
How can I improve the robustness against lighting?
Update
I'm experimenting with a solution that works quite well, but it need some fixes.
I'm using MOG2 in this manner:
Acquiring and learning background using the first frames (BGK)
Apply MOG2 to current frame with learning rate of 0 (no update) and get foreground mask (FG_MASK)
For the next frames I'm using FG_MASK to mask BGK and I'm using the result to Apply to MOG2 with some learning rate (this update the background).
After that I'm updating BGK taking it from MOG2 algorythm.
In this way, objects are masked out of the background, and the background still updating. This can guarantee a good robustness against light changes.
There is some drawback, for example when the light is changing, the object mask ("mask blob") keep with the previous brightness, and if the difference is too high can be detected as new object.
In the above image you can see that the current frame is brighter and the mask for the static object is darker.
My idea is try to adapt the "mask blob" changing it's brightness following the light changing. How can i get this with OpenCV?
Fix for previous drawbacks
Using inpaint function instaead to simply mask the BGK (step 3) i can keep the "mask blobs" sync with background brightness changes.
This fix has drawback too, it's not very perfomance.
Update 2
I think this is an interesting topic so I keep it updated.
The inpaint function is very slow, so I'm trying another way.
Using the Hsv color space allows you to manage the brightness channel, I can reduce the impact of brightness in this way:
obtain the V channel with the Split function
calculate the mean value of channel V
apply a threshold truncate to the V channel using the mean value
Rebuild frame using new V channel
I had similar problem implementing speed estimation algorithm, I hope that my solution may help you.
One of the methods I tried was Accumulative Difference Image (basically what you did with MOG2), but it failed to track stationary objects when the background updated. When I did not update the background I had the same problem as you did.
So, I decided to use RGB/HSV thresholding. I set the boundaries for color of the road (let us say gray), and created binary image, where everything of a color of the road was black (0), everything else was white (1). Here is a nice tutorial on HSV threshold. When choosing boundaries you can acknowledge the lighting factor setting let us say upper boundary for bright lighting and lower for dark. However, this method may cause object of a color similar to background be not seen by the algorithm. Another shortcoming is the the background should be uniform, without any details.
Another method you can try is to convert both input image and background to grayscale and then subtract manually. This would give you an opportunity to tweak the threshold level for difference from the background. Let us say background of a value 120 in dark condition will have 140 in bright condition, so difference is 20. For object pixel may have let us say value of 180 and background value is 120, so difference is 60. Set threshold for difference 20 and set values below 20 to 0 and values above 20 to 1, this should do the thing (all values are on the scale from 0 to 255).
Good luck!
Background:
Im pulsing an infrared IR LED on/off and capturing the images with an USB camera. For example: In the odd frames the LED is on and in the even frames the LED is off (I cant control in which frames it is on/off). Now Im subtracting the current frame with the last using cvAbsDiff. Usually only the LED signal remains in the image.
Now the problem:
When there is strong ambient light like it is coming from the sun, it creates a strong shadow. When I subtract the current frame with the last frame and move any object fast enough, this object is not at the same position in both frames. So the frame-to-frame subtraction will give me the edges of this moved object (the motion) and will interfere with the LED signal.
How can I remove the "motion" from the resulting image and preserve the pulsing LED?
Note: I tried using cvThreshold. The intensities between LED and the motion is almost the same.