Programming with the Kinect for Windows Software Development Kit

David Catuhe

Mentioned 2

Create rich experiences for users of Windows 7 and Windows 8 Developer Preview with this pragmatic guide to the Kinect for Windows Software Development Kit (SDK). The author, a developer evangelist for Microsoft, walks you through Kinect sensor technology and the SDK--providing hands-on insights for how to add gesture and posture recognition to your apps. If you're skilled in C# and Windows Presentation Foundation, you'll learn how to integrate Kinect in your applications and begin writing Uis and controls that can handle Kinect interaction. This book introduces the Kinect for Windows Software Development Kit to developers looking to enrich applications they build for Windows 7 and later with human motion tracking Teaches developers with core C# and WPF skills how to program gesture and posture recognition in Kinect Describes how to integrate 3D representation on top of a real scene Provides expert insights and code samples to get you up and running

More on Amazon.com

Mentioned in questions and answers.

I am using an XBox Kinect with the Kinect for Windows SDK. I want to make an application that will augment a 3D mask (a 3D model of a mask made in 3DS Max) onto the face of anyone using the application. The application will be used in an exhibit locally. I have not tried much because I don't know where to start. So what I want to know is, is it currently possible to augment a 3DS Max model onto a live video stream using the facial recognition and skeletal tracking features in the newest Kinect for Windows SDK, and if so, how/where should I start trying to do/implement this? Any point in the right direction would be great. Thank you! PS And yes, I have read the UI guidelines and the facial documentation. My problem is one of not knowing where to start programming, not one of not understanding the fundamental concepts. Thanks!

If you are serious about getting into developing for the Kinect I would recommend getting this book: http://www.amazon.com/Programming-Kinect-Windows-Software-Development/dp/0735666814

This goes through developing with the Kinect for Windows SDK from the ground up. There is a face tracking and an augmented reality example so I'm pretty sure you will be able to achieve your goal quite easily.

All the code from the book is here: http://kinecttoolbox.codeplex.com/


Alternatively, there is an example here which pretty much is what you want to achieve: http://www.codeproject.com/Articles/213034/Kinect-Getting-Started-Become-The-Incredible-Hulk

It is developed using the Beta version of the SDK, but the same priciples apply.


You can also check out the quick start videos here:

http://channel9.msdn.com/Series/KinectQuickstart


In summary, based on my own experience, I would spend some time going through the beginner examples either in the vides or the book (I found the book very good) just to get familiar with how to setup a simple Kinect project and how the different parts of the SDK work.

When you have developed some throwaway apps with the Kinect, I would then try tackling your project (although, the Incredible Hulk project above should get you most the way there!)

Best of luck with your project

I am writing a Kinect-based application. I need to implement a wave-right gesture. I found a code snippet from the Internet using C#, but I have some doubts about this code. Can you help me?

The code:

public class KinectClass
{
    public void KinectInitialize ()
    {
        m_myKinect.Start();
        m_myKinect.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
        m_myKinect.SkeletonFrameReady += m_myKinect_SkeletonFrameReady;
    }

    void m_myKinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
    {
        using (SkeletonFrame frame = e.OpenSkeletonFrame())
        {
            if (frame != null)
            {
                if (0 == frame.SkeletonArrayLength) return;
                m_skeletons = new Skeleton[frame.SkeletonArrayLength];
                if (m_skeletons == null) return;
                frame.CopySkeletonDataTo(m_skeletons);
            }
        }

        foreach (Skeleton skeleton in m_skeletons)
        {
            if (skeleton.TrackingState == SkeletonTrackingState.Tracked)
            {
                UpdateSkeletonHistory(skeleton);
                value = gestureobject.Detectgesture();
            }
        }
    }
}

In addition, there is a Gesture class which is checking wave functions:

class Gesture
{
    private List<Skeleton> SkeletonHistory;
    private List<Skeleton> localSkeletonHistory;

    public void UpdateSkeletonHistory(Skeleton SkeletonData)
    {
        SkeletonHistory.Add(SkeletonData);

        if (SkeletonHistory.Count > 90)
        {
            SkeletonHistory.RemoveAt(0);
        }

        return ;
    }

    public short Detectgesture()
    {
        short sReturnvalue = -1;
        int gesturePeriod = 2;
        int indexesPerSecond = 30;
        int indexesToCheck = gesturePeriod * indexesPerSecond;
        localSkeletonHistory = new List<Skeleton>(SkeletonHistory);

        if (m_SkeletonHistory.Count > indexesToCheck)
        {
            localSkeletonHistory = new List<Skeleton>(m_SkeletonHistory);

            if( waveright() )
            {
                Console.Writeline("waveright");
            }
        }
    }

    private bool waveright()
    {
        Console.WriteLine("wave right START");
        bool Movedright = false,
        Movedleft = false;

        float refDistance = 0.2F;

        float startPos = localSkeletonHistory[0].Joints[JointType.HandLeft].Position.X;

        for (int i = 0; i < localSkeletonHistory.Count; i++)
        {
            if (!(localSkeletonHistory[i].Joints[JointType.HandLeft].Position.Y <= localSkeletonHistory[i].Joints[JointType.Head].Position.Y))
            {
                break;
            }
            if (localSkeletonHistory[i].Joints[JointType.HandLeft].Position.X >= (startPos + refDistance))
            {
                Movedright = true;
            }
            if (Movedright && Math.Abs(localSkeletonHistory[i].Joints[JointType.HandLeft].Position.X - startPos) <= 0.1 )
            {
                Movedleft = true;
                m_SkeletonHistory.Clear();
                break;
            }
        }

        return Movedleft;
    }
}

I can understand Kinect functions, but I am confused about the gesture functions. My questions are as follows:

  • Why do they create a SkeletonHistory?
  • What is the need for gesturePeriod and indexesPerSecond?
  • Why do they check if (m_SkeletonHistory.Count > indexesToCheck)?
  • Can you explain the waveright function?

The thing you need to remember is that just because the Hand's position is right, relative to the body. It does not mean that they are waving right.

A wave right is a transition towards the right. So you cannot determine that by the current position. That is why you need to keep track of the previous positions using SkeletonHistory.

So a blunt overview of the algorithm is:

For the previous positions since the last triggered gesture:
     If each Skeleton's Y position is greater than the last, then the person is waving right

Of course, a hand will move right many times, it does not mean it is a wave right. So you need to determine how long a hand must be moving right, for it to be a genuine waveRight gesture.

refDistance is a way of saying that the hand needed to move at least this distance before it can be categorised as a wave right.

Here, I believe the coder is saying that the gesture needs to last at least 2 seconds. So if we get 30 skeletons per second, then we need to check the last 60 skeletons:

int indexesToCheck = gesturePeriod * indexesPerSecond;

The msdn documentation is actually very good for this stuff:

https://msdn.microsoft.com/en-us/library/jj131025.aspx

I found this book excellent for getting started on this stuff:

Programming with the Kinect for Windows Software Development Kit (Developer Reference)