Trouble With Kinect Skeleton Tracking in F# - f#

Question

I am using the F# skeleton tracking template provided by KinectContrib. The template in C# that does the same thing works so I know the hardware is OK.
I am using Windows Kinect SDK v1.8.
The program will track once in a rare while but with no consistent pattern. I have been playing with the code since last night so I am looking for someone to confirm the same behavior on another system or for any pointers on how to change the code.
Thanks in advance.
This is the template code:
#light
open System
open System.Windows
open System.Windows.Media.Imaging
open Microsoft.Kinect
open System.Diagnostics
let sensor = KinectSensor.KinectSensors.[0]
//The main canvas that is handling the ellipses
let canvas = new System.Windows.Controls.Canvas()
canvas.Background <- System.Windows.Media.Brushes.Transparent
let ds : byte = Convert.ToByte(1)
let dummySkeleton : Skeleton = new Skeleton(TrackingState = SkeletonTrackingState.Tracked)
// Thanks to Richard Minerich (#rickasaurus) for helping me figure out
// some array concepts in F#.
let mutable pixelData : byte array = [| |]
let mutable skeletons : Skeleton array = [| |]
//Right hand ellipse
let rhEllipse = new System.Windows.Shapes.Ellipse()
rhEllipse.Height <- 20.0
rhEllipse.Width <- 20.0
rhEllipse.Fill <- System.Windows.Media.Brushes.Red
rhEllipse.Stroke <- System.Windows.Media.Brushes.White
//Left hand ellipse
let lhEllipse = new System.Windows.Shapes.Ellipse()
lhEllipse.Height <- 20.0
lhEllipse.Width <- 20.0
lhEllipse.Fill <- System.Windows.Media.Brushes.Red
lhEllipse.Stroke <- System.Windows.Media.Brushes.White
//Head ellipse
let hEllipse = new System.Windows.Shapes.Ellipse()
hEllipse.Height <- 20.0
hEllipse.Width <- 20.0
hEllipse.Fill <- System.Windows.Media.Brushes.Red
hEllipse.Stroke <- System.Windows.Media.Brushes.White
canvas.Children.Add(rhEllipse) |> ignore
canvas.Children.Add(lhEllipse) |> ignore
canvas.Children.Add(hEllipse) |> ignore
let grid = new System.Windows.Controls.Grid()
let winImage = new System.Windows.Controls.Image()
winImage.Height <- 600.0
winImage.Width <- 800.0
grid.Children.Add(winImage) |> ignore
grid.Children.Add(canvas) |> ignore
//Video frame is ready to be processed.
let VideoFrameReady (sender : obj) (args: ColorImageFrameReadyEventArgs) =
let receivedData = ref false
using (args.OpenColorImageFrame()) (fun r ->
if (r <> null) then
(
pixelData <- Array.create r.PixelDataLength ds
//Array.Resize(ref pixelData, r.PixelDataLength)
r.CopyPixelDataTo(pixelData)
receivedData := true
)
if (receivedData <> ref false) then
(
winImage.Source <- BitmapSource.Create(640, 480, 96.0, 96.0, Media.PixelFormats.Bgr32, null, pixelData, 640 * 4)
)
)
//Required to correlate the skeleton data to the PC screen
//IMPORTANT NOTE: Code for vector scaling was imported from the Coding4Fun Kinect Toolkit
//available here: http://c4fkinect.codeplex.com/
//I only used this part to avoid adding an extra reference.
let ScaleVector (length : float32, position : float32) =
let value = (((length / 1.0f) / 2.0f) * position) + (length / 2.0f)
if value > length then
length
elif value < 0.0f then
0.0f
else
value
//This will set the ellipse positions depending on the passed instance and joint
let SetEllipsePosition (ellipse : System.Windows.Shapes.Ellipse, joint : Joint) =
let vector = new Microsoft.Kinect.SkeletonPoint(X = ScaleVector(640.0f, joint.Position.X), Y=ScaleVector(480.0f, -joint.Position.Y),Z=joint.Position.Z)
let mutable uJoint = joint
uJoint.TrackingState <- JointTrackingState.Tracked
uJoint.Position <- vector
System.Windows.Controls.Canvas.SetLeft(ellipse,(float uJoint.Position.X))
System.Windows.Controls.Canvas.SetTop(ellipse,(float uJoint.Position.Y))
//Triggered when a new skeleton frame is ready for processing
let SkeletonFrameReady (sender : obj) (args: SkeletonFrameReadyEventArgs) =
let receivedData = ref false
using (args.OpenSkeletonFrame()) (fun r ->
if (r <> null) then
(
skeletons <- Array.create r.SkeletonArrayLength dummySkeleton
r.CopySkeletonDataTo(skeletons)
for i in skeletons do
Debug.WriteLine(i.TrackingState.ToString())
receivedData := true
)
if (receivedData <> ref false) then
(
for i in skeletons do
if i.TrackingState <> SkeletonTrackingState.NotTracked then
(
let currentSkeleton = i
SetEllipsePosition(hEllipse, currentSkeleton.Joints.[JointType.Head])
SetEllipsePosition(lhEllipse, currentSkeleton.Joints.[JointType.HandLeft])
SetEllipsePosition(rhEllipse, currentSkeleton.Joints.[JointType.HandRight])
)
)
)
let WindowLoaded (sender : obj) (args: RoutedEventArgs) =
sensor.Start()
sensor.ColorStream.Enable()
sensor.SkeletonStream.Enable()
sensor.ColorFrameReady.AddHandler(new EventHandler<ColorImageFrameReadyEventArgs>(VideoFrameReady))
sensor.SkeletonFrameReady.AddHandler(new EventHandler<SkeletonFrameReadyEventArgs>(SkeletonFrameReady))
let WindowUnloaded (sender : obj) (args: RoutedEventArgs) =
sensor.Stop()
//Defining the structure of the test window
let window = new Window()
window.Width <- 800.0
window.Height <- 600.0
window.Title <- "Kinect Skeleton Application"
window.Loaded.AddHandler(new RoutedEventHandler(WindowLoaded))
window.Unloaded.AddHandler(new RoutedEventHandler(WindowUnloaded))
window.Content <- grid
window.Show()
[<STAThread()>]
do
let app = new Application() in
app.Run(window) |> ignore

Answer 1

I ended up rewriting it based off of this post http://channel9.msdn.com/coding4fun/kinect/Kinecting-with-F and the skeleton tracking is now working. Still interested in why the original code doesn't work as well.
// Learn more about F# at http://fsharp.net
#light
open System
open System.Windows
open System.Windows.Media.Imaging
open System.Windows.Threading
open Microsoft.Kinect
open System.Diagnostics
[<STAThread>]
do
let sensor = KinectSensor.KinectSensors.[0]
sensor.SkeletonStream.Enable()
sensor.Start()
// Set-up the WPF window and its contents
let width = 1024.
let height = 768.
let w = Window(Width=width, Height=height)
let g = Controls.Grid()
let c = Controls.Canvas()
let hd = Shapes.Rectangle(Fill=Media.Brushes.Red, Width=15., Height=15.)
let rh = Shapes.Rectangle(Fill=Media.Brushes.Blue, Width=15., Height=15.)
let lh = Shapes.Rectangle(Fill=Media.Brushes.Green, Width=15., Height=15.)
ignore <| c.Children.Add hd
ignore <| c.Children.Add rh
ignore <| c.Children.Add lh
ignore <| g.Children.Add c
w.Content <- g
w.Unloaded.Add(fun args -> sensor.Stop())
let getDisplayPosition w h (joint : Joint) =
let x = ((w * (float)joint.Position.X + 2.0) / 4.0) + (w/2.0)
let y = ((h * -(float)joint.Position.Y + 2.0) / 4.0) + (h/2.0)
System.Console.WriteLine("X:" + x.ToString() + " Y:" + y.ToString())
new Point(x,y)
let draw (joint : Joint) (sh : System.Windows.Shapes.Shape) =
let p = getDisplayPosition width height joint
sh.Dispatcher.Invoke(DispatcherPriority.Render, Action(fun () -> System.Windows.Controls.Canvas.SetLeft(sh, p.X))) |> ignore
sh.Dispatcher.Invoke(DispatcherPriority.Render, Action(fun () -> System.Windows.Controls.Canvas.SetTop(sh, p.Y))) |> ignore
let drawJoints (sk : Skeleton) =
draw (sk.Joints.Item(JointType.Head)) hd
draw (sk.Joints.Item(JointType.WristRight)) rh
draw (sk.Joints.Item(JointType.WristLeft)) lh
let skeleton (sensor : KinectSensor) =
let rec loop () =
async {
let! args = Async.AwaitEvent sensor.SkeletonFrameReady
use frame = args.OpenSkeletonFrame()
let skeletons : Skeleton[] = Array.zeroCreate(frame.SkeletonArrayLength)
frame.CopySkeletonDataTo(skeletons)
skeletons
|> Seq.filter (fun s -> s.TrackingState <> SkeletonTrackingState.NotTracked)
|> Seq.iter (fun s -> drawJoints s)
return! loop ()
}
loop ()
skeleton sensor |> Async.Start
let a = Application()
ignore <| a.Run(w)

Answer 2

In F#, any value bindings (e.g., let or do) you declare within a module itself will be executed the first time the module is opened or accessed from another module. If you're familiar with C#, you can think of these value bindings as executing within a type constructor (i.e., a static constructor).
I suspect the reason the second version of your code works, but not the first, is that in the second version, you're creating the Window and drawing the shapes into it from within the STA thread running the application's message loop. In the first version, I'd guess that code is executing on some other thread, and that's why it isn't working as expected.
There's nothing wrong with the second version of your code, but a more-canonical F# approach would be to lift your functions (getDisplayPosition, draw, etc.) out of the top-level do binding. That makes the code a bit easier to read by making it obvious that those functions aren't capturing any of the local values created within the do.