I'm currently working on a project where I need to be able to very reliable get the positions of the balls on a pool table.
I'm using a Kinect v2 above the table as the source.
Initial image looks like this (after converting it to 8-bit from 16-bit by throwing away pixels which is not around table level):
Then a I subtract a reference image with the empty table from the current image.
After thresholding and equalization it looks like this: image
It's fairly easy to detect the individual balls on a single image, the problem is that I have to do it constantly with 30fps.
Difficulties:
Low resolution image (512*424), a ball is around 4-5 pixel in diameter
Kinect depth image has a lot of noise from this distance (2 meters)
Balls look different on the depth image, for example the black ball is kind of inverted compared to the others
If they touch each other then they can become one blob on the image, if I try to separate them with depth thresholding (only using the top of the balls) then some of the balls can disappear from the image
It's really important that anything other than balls should not be detected e.g.: cue, hands etc...
My process which kind of works but not reliable enough:
16bit to 8bit by thresholding
Subtracting sample image with empty table
Cropping
Thresholding
Equalizing
Eroding
Dilating
Binary threshold
Contour finder
Some further algorithms on the output coordinates
The problem is that a pool cue or hand can be detected as a ball and also if two ball touches then it can cause issues. Also tried with hough circles but with even less success. (Works nicely if the Kinect is closer but then it cant cover the whole table)
Any clues would be much appreciated.
Expanding comments above:
I recommend improving the IRL setup as much as possible.
Most of the time it's easier to ensure a reliable setup than to try to "fix" that user computer vision before even getting to detecting/tracking anything.
My suggestions are:
Move the camera closer to the table. (the image you posted can be 117% bigger and still cover the pockets)
Align the camera to be perfectly perpendicular to the table (and ensure the sensor stand is sturdy and well fixed): it will be easier to process a perfect top down view than a slightly tilted view (which is what the depth gradient shows). (sure the data can be rotated, but why waste CPU cycles when you can simply keep the sensor straight)
With a more reliable setup you should be able to threshold based on depth.
You can possible threshold to the centre of balls since the information bellow is occluded anyway. The balls do not deform, so it the radius decreases fast the ball probably went in a pocket.
One you have a clear threshold image, you can findContours() and minEnclosingCircle(). Additionally you should contrain the result based on min and max radius values to avoid other objects that may be in the view (hands, pool cues, etc.). Also have a look at moments() and be sure to read Adrian's excellent Ball Tracking with OpenCV article
It's using Python, but you should be able to find OpenCV equivalent call for the language you use.
In terms tracking
If you use OpenCV 2.4 you should look into OpenCV 2.4's tracking algorithms (such as Lucas-Kanade).
If you already use OpenCV 3.0, it has it's own list of contributed tracking algorithms (such as TLD).
I recommend starting with Moments first: use the simplest and least computationally expensive setup initially and see how robuts the results are before going into the more complex algorithms (which will take to understand and get the parameters right to get expected results out of)
I am creating a game and I need to draw some obstacles on in (SKNodes), depending on the level (background image).
Let's say I have a tree, i would like to draw borders around it as in the picture below:
I will need the coordinates to create a physicsBody.
Is there a simple way to do it? (maybe a software?)
I will need to adapt the same coordinates for different devices, therefore I believe it's too complicated to manually do it.
Any ideas?
Thanks!
There is a SKPhysicsBody class method that could be of help:
+ bodyWithTexture:size:
From the description:
Use this method when your sprite has a shape that you want replicated
in its physics body. The texture is scaled to the new size and then
analyzed. A new physics body is created that includes all of the
texels in the texture that have a nonzero alpha value. The shape of
this body attempts to strike a good balance between performance and
accuracy. For example, fine details may be ignored if keeping them
would cause a significant performance penalty.
Check out this SKPhysicsBody Path Generator.
You can easily create a path as you desire with this tool (but make sure you read the rules at the top right)
My question maybe a bit too broad but i am going for the concept. How can i create surface as they did in "Cham Cham" app
https://itunes.apple.com/il/app/cham-cham/id760567889?mt=8.
I got most of the stuff done in the app but the surface change with user touch is quite different. You can change its altitude and it grows and shrinks. How this can be done using sprite kit what is the concept behind that can anyone there explain it a bit.
Thanks
Here comes the answer from Cham Cham developers :)
Let me split the explanation into different parts:
Note: As the project started quite a while ago, it is implemented using pure OpenGL. The SpiteKit implementation might differ, but you just need to map the idea over to it.
Defining the ground
The ground is represented by a set of points, which are interpolated over using Hermite Spline. Basically, the game uses a bunch of points defining the surface, and a set of points between each control one, like the below:
The red dots are control points, and eveyrthing in between is computed used the metioned Hermite interpolation. The green points in the middle have nothing to do with it, but make the whole thing look like boobs :)
You can choose an arbitrary amount of steps to make your boobs look as smooth as possible, but this is more to do with performance.
Controlling the shape
All you need to do is to allow the user to move the control points (or some of them, like in Cham Cham; you can define which range every point could move in etc). Recomputing the interpolated values will yield you an changed shape, which remains smooth at all times (given you have picked enough intermediate points).
Texturing the thing
Again, it is up to you how would you apply the texture. In Cham Cham, we use one big texture to hold the background image and recompute the texture coordinates at every shape change. You could try a more sophisticated algorithm, like squeezing the texture or whatever you found appropriate.
As for the surface texture (the one that covers the ground – grass, ice, sand etc) – you can just use the thing called Triangle Strips, with "bottom" vertices sitting at every interpolated point of the surface and "top" vertices raised over (by offsetting them against "bottom" ones in the direction of the normal to that point).
Rendering it
The easiest way is to utilize some tesselation library, like libtess. What it will do it covert you boundary line (composed of interpolated points) into a set of triangles. It will preserve texture coordinates, so that you can just feed these triangles to the renderer.
SpriteKit note
Unfortunately, I am not really familiar with SpriteKit engine, so cannot guarantee you will be able to copy the idea over one-to-one, but please feel free to comment on the challenging aspects of the implementation and I will try to help.
Given one texture sheet is it better to have one or multiple CCSpriteBatchNodes? Or does this not affect at all the GPU computational cost in processing the non visible CCSprite quads?
I am thinking about performance and referring to this question and answer I got. Basically it suggests that I should use more than one CCSpriteBatchNode even if I have only one file. I don't understand if the sentence "Too many batched sprites still affects performance negatively even if they are not visible or outside the screen" is applicable also having two CCSpriteBatchNode instead of one. In other words, does the sentence refer to this "The GPU is responsible for cancelling draws of quads that are not visible due to being entirely outside the screen. It still needs to process those quads."? And if so it should meant that it doesn't really matter how may CCSpriteBatchNode instances I have using the same texture sheet, right?
How can I optimize this? I mean, how can I avoid the GPU having to process the non visible quads?
Would you be able to answer to at least the questions in bold?
First case: Too many nodes (or sprites) in the scene and many of them are out of screen/visible area. In this case for each sprite, GPU has to check if its outside the visible area or not. Too many sprite-nodes means too much load on GPU.
Adding more CCSpriteBatchNode should not effect the performance. Because the sprite-sheet bitmap is loaded to the GPU memory, and an array of coordinates is kept by the application for drawing individual sprites. So if you put 2 images in 2 different CCSpriteBatchNodes or 2 images in 1, it will be same for both CPU and GPU.
How to optimize?
The best way would be to remove the invisible nodes/sprites from the parent. But it depends on your application.
FPS drops certainly because of two reasons:
fillrate - when a lot of sprites overlap each others (and additionally if we render high-res texture into small sprite)
redundant state changes - in this case the heaviest are shader and texture switches
You can render sprites outside of screen in single batch and this doesn't drop performance singnificantly. Pay attention that rendering sprite with zero opacity (or transparent texture) takes the same time as non-transparent sprite.
First of all, this really sounds like a case of premature optimization. Do a test with the number of sprites you expect to be on screen, and some added, others removed. Do you get 60 fps on the oldest supported device? If yes, good, no need to optimize. If no, tweak the code design to see what actually makes a difference.
I mean, how can I avoid the GPU having to process the non visible quads?
You can't, unless you're going to rewrite how cocos2d handles drawing of sprites/batched sprites.
it doesn't really matter how may CCSpriteBatchNode instances I have using the same texture sheet, right?
Each additional sprite batch node adds a draw call. Given how many sprites they can batch into a single draw call, the benefit far outweighs the drawbacks. Whether you have one, two or three sprite batch nodes makes absolutely no difference.
I'm working on a 2D Platform game, and I was wondering what's the best (performance-wise) way to implement Surface (Collision) Detection.
So far I'm thinking of constructing a list of level objects constructed of a list of lines, and I draw tiles along the lines.
alt text http://img375.imageshack.us/img375/1704/lines.png
I'm thinking every object holds the ID of the surface that he walks on, in order to easily manipulate his y position while walking up/downhill.
Something like this:
//Player/MovableObject class
MoveLeft()
{
this.Position.Y = Helper.GetSurfaceById(this.SurfaceId).GetYWhenXIs(this.Position.X)
}
So the logic I use to detect "droping/walking on surface" is a simple point (player's lower legs)-touches-line (surface) check
(with some safety approximation
- let`s say 1-2 pixels over the line).
Is this approach OK?
I`ve been having difficulty trying to find reading material for this problem, so feel free to drop links/advice.
Having worked with polygon-based 2D platformers for a long time, let me give you some advice:
Make a tile-based platformer.
Now, to directly answer your question about collision-detection:
You need to make your world geometry "solid" (you can get away with making your player object a point, but making it solid is better). By "solid" I mean - you need to detect if the player object is intersecting your world geometry.
I've tried "does the player cross the edge of this world geometry" and in practice is doesn't work (even though it might seem to work on paper - floating point precision issues will not be your only problem).
There are lots of instructions online on how to do intersection tests between various shapes. If you're just starting out I recommend using Axis-Aligned Bounding Boxes (AABBs).
It is much, much, much, much, much easier to make a tile-based platformer than one with arbitrary geometry. So start with tiles, detect intersections with AABBs, and then once you get that working you can add other shapes (such as slopes).
Once you detect an intersection, you have to perform collision response. Again a tile-based platformer is easiest - just move the player just outside the tile that was collided with (do you move above it, or to the side? - it will depend on the collision - I will leave how to do this is an exercise).
(PS: you can get terrific results with just square tiles - look at Knytt Stories, for example.)
Check out how it is done in the XNA's Platformer Starter Kit Project. Basically, the tiles have enum for determining if the tile is passable, impassable etc, then on your level you GetBounds of the tiles and then check for intersections with the player and determine what to do.
I've had wonderful fun times dealing with 2D collision detection. What seems like a simple problem can easily become a nightmare if you do not plan it out in advance.
The best way to do this in a OO-sense would be to make a generic object, e.g. classMapObject. This has a position coordinate and slope. From this, you can extend it to include other shapes, etc.
From that, let's work with collisions with a Solid object. Assuming just a block, say 32x32, you can hit it from the left, right, top and bottom. Or, depending on how you code, hit it from the top and from the left at the same time. So how do you determine which way the character should go? For instance, if the character hits the block from the top, to stand on, coded incorrectly you might inadvertently push the character off to the side instead.
So, what should you do? What I did for my 2D game, I looked at the person's prior positioning before deciding how to react to the collision. If the character's Y position + Height is above the block and moving west, then I would check for the top collision first and then the left collision. However, if the Character's Y position + height is below the top of the block, I would check the left collision.
Now let's say you have a block that has incline. The block is 32 pixels wide, 32 pixels tall at x=32, 0 pixels tall at x=0. With this, you MUST assume that the character can only hit and collide with this block from the top to stand on. With this block, you can return a FALSE collision if it is a left/right/bottom collision, but if it is a collision from the top, you can state that if the character is at X=0, return collision point Y=0. If X=16, Y=16 etc.
Of course, this is all relative. You'll be checking against multiple blocks, so what you should do is store all of the possible changes into the character's direction into a temporary variable. So, if the character overlaps a block by 5 in the X direction, subtract 5 from that variable. Accumulate all of the possible changes in the X and Y direction, apply them to the character's current position, and reset them to 0 for the next frame.
Good luck. I could provide more samples later, but I'm on my Mac (my code is on a WinPC) This is the same type of collision detection used in classic Mega Man games IIRC. Here's a video of this in action too : http://www.youtube.com/watch?v=uKQM8vCNUTM
You can try to use one of physics engines, like Box2D or Chipmunk. They have own advanced collision detection systems and a lot of different bonuses. Of course they don't accelerate your game, but they are suitable for most of games on any modern devices
It is not that easy to create your own collision detection algorithm. One easy example of a difficulty is: what if your character is moving at a high enough velocity that between two frames it will travel from one side of a line to the other? Then your algorithm won't have had time to run in between, and a collision will never be detected.
I would agree with Tiendil: use a library!
I'd recommend Farseer Physics. It's a great and powerful physics engine that should be able to take care of anything you need!
I would do it this way:
Strictly no lines for collision. Only solid shapes (boxes and triangles, maybe spheres)
2D BSP, 2D partitioning to store all level shapes, OR "sweep and prune" algorithm. Each of those will be very powerfull. Sweep and prune, combined with insertion sort, can easily thousands of potentially colliding objects (if not hundreds of thousands), and 2D space partitioning will allow to quickly get all nearby potentially colliding shapes on demand.
The easiest way to make objects walk on surfaces is to make then fall down few pixels every frame, then get the list of surfaces object collides with, and move object into direction of surface normal. In 2d it is a perpendicular. Such approach will cause objects to slide down on non-horizontal surfaces, but you can fix this by altering the normal slightly.
Also, you'll have to run collision detection and "push objects away" routine several times per frame, not just once. This is to handle situations if objects are in a heap, or if they contact multiple surfaces.
I have used a limited collision detection approach that worked on very different basis so I'll throw it out here in case it helps:
A secondary image that's black and white. Impassible pixels are white. Construct a mask of the character that's simply any pixels currently set. To evaluate a prospective move read the pixels of that mask from the secondary image and see if a white one comes back.
To detect collisions with other objects use the same sort of approach but instead of booleans use enough depth to cover all possible objects. Draw each object to the secondary entirely in the "color" of it's object number. When you read through the mask and get a non-zero pixel the "color" is the object number you hit.
This resolves all possible collisions in O(n) time rather than the O(n^2) of calculating interactions.