Programming with the Kinect for Windows Software Development Kit: Algorithmic Gestures and Postures

9/15/2012

Contents

Defining a gesture with an algorithm
Defining a posture with an algorithm

Defining a posture with an algorithm

To detect simple postures, it is possible to track distances, relative positions, or angles between given joints. For example, to detect a “hello” posture, you have to check to determine if one hand is higher than the head and at the same time check to make sure the x and z coordinates are not too far from each other. For the “hands joined” posture, you must check to find out if the positions of the two hands are almost the same.

Creating a base class for posture detection

Using the same concepts that you used to define gestures, you can write an abstract base class for detecting postures. This class provides a set of services for children classes:

An event to signal detected postures
A solution to handle the stability of the posture

Unlike gestures, however, postures cannot be detected immediately, because to guarantee that the posture is a wanted posture, the system must check that the posture is held for a defined number of times.

The PostureDetector class is then defined as follows:

using System;
using Microsoft.Kinect;

namespace Kinect.Toolbox
{
    public abstract class PostureDetector
    {
        public event Action<string> PostureDetected;

        readonly int accumulatorTarget;
        string previousPosture = "";
        int accumulator;
        string accumulatedPosture = "";

        public string CurrentPosture
        {
            get { return previousPosture; }
            protected set { previousPosture = value; }
        }

        protected PostureDetector(int accumulators)
        {
            accumulatorTarget = accumulators;
        }

        public abstract void TrackPostures(Skeleton skeleton);

        protected void RaisePostureDetected(string posture)
        {
            if (accumulator < accumulatorTarget)
            {
                if (accumulatedPosture != posture)
                {
                    accumulator = 0;
                    accumulatedPosture = posture;
                }
                accumulator++;
                return;
            }

            if (previousPosture == posture)
                return;

            previousPosture = posture;
            if (PostureDetected != null)
                PostureDetected(posture);

            accumulator = 0;
        }

        protected void Reset()
        {
            previousPosture = "";
            accumulator = 0;
        }
    }
}

The accumulatorTarget property is used to define how many times a posture must be detected before it can be signaled to user.

To use the class, the user simply has to call TrackPostures with a skeleton. Children classes provide implementation for this method and will call RaisePostureDetected when a posture is found. RaisePostureDetected counts the number of times a given posture (previousPosture) is detected and raises the PostureDetected event only when accumulatorTarget is met.

Detecting simple postures

Inheriting from PostureDetector, you can now create a simple class responsible for detecting common simple postures. This class has to track given joints positions and accordingly can raise PostureDetected.

The code is as follows:

using System;
using Microsoft.Kinect;

namespace Kinect.Toolbox
{
    public class AlgorithmicPostureDetector : PostureDetector
    {
        public float Epsilon {get;set;}
        public float MaxRange { get; set; }

        public AlgorithmicPostureDetector() : base(10)
        {
            Epsilon = 0.1f;
            MaxRange = 0.25f;
        }

        public override void TrackPostures(Skeleton skeleton)
        {
            if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
                return;

            Vector3? headPosition = null;
            Vector3? leftHandPosition = null;
            Vector3? rightHandPosition = null;

            foreach (Joint joint in skeleton.Joints)
            {
                if (joint.TrackingState != JointTrackingState.Tracked)
                    continue;

                switch (joint.JointType)
                {
                    case JointType.Head:
                        headPosition = joint.Position.ToVector3();
                        break;
                    case JointType.HandLeft:
                        leftHandPosition = joint.Position.ToVector3();
                        break;
                    case JointType.HandRight:
                        rightHandPosition = joint.Position.ToVector3();
                        break;
                }
            }

            // HandsJoined
            if (CheckHandsJoined(rightHandPosition, leftHandPosition))
            {
                RaisePostureDetected("HandsJoined");
                return;
            }

            // LeftHandOverHead
            if (CheckHandOverHead(headPosition, leftHandPosition))
            {
                RaisePostureDetected("LeftHandOverHead");
                return;
            }

            // RightHandOverHead
            if (CheckHandOverHead(headPosition, rightHandPosition))
            {
                RaisePostureDetected("RightHandOverHead");
                return;
            }

            // LeftHello
            if (CheckHello(headPosition, leftHandPosition))
            {
                RaisePostureDetected("LeftHello");
                return;
            }

            // RightHello
            if (CheckHello(headPosition, rightHandPosition))
            {
                RaisePostureDetected("RightHello");
                return;
            }

            Reset();
        }

        bool CheckHandOverHead(Vector3? headPosition, Vector3? handPosition)
        {
            if (!handPosition.HasValue || !headPosition.HasValue)
                return false;

            if (handPosition.Value.Y < headPosition.Value.Y)
                return false;

            if (Math.Abs(handPosition.Value.X - headPosition.Value.X) > MaxRange)
                return false;

            if (Math.Abs(handPosition.Value.Z - headPosition.Value.Z) > MaxRange)
                return false;

            return true;
        }


        bool CheckHello(Vector3? headPosition, Vector3? handPosition)
        {
            if (!handPosition.HasValue || !headPosition.HasValue)
                return false;

            if (Math.Abs(handPosition.Value.X - headPosition.Value.X) < MaxRange)
                return false;

            if (Math.Abs(handPosition.Value.Y - headPosition.Value.Y) > MaxRange)
                return false;

            if (Math.Abs(handPosition.Value.Z - headPosition.Value.Z) > MaxRange)
                return false;

            return true;
        }

        bool CheckHandsJoined(Vector3? leftHandPosition, Vector3? rightHandPosition)
        {
            if (!leftHandPosition.HasValue || !rightHandPosition.HasValue)
                return false;

            float distance = (leftHandPosition.Value - rightHandPosition.Value).Length;

            if (distance > Epsilon)
                return false;

            return true;
        }
    }
}

As you can see, the class only tracks hands and head positions. (To be sure, only tracked joints are taken into account.) With these positions, a group of methods (CheckHandOverHead, CheckHello, CheckHandsJoined) are called to detect specific postures.

Consider CheckHandOverHead:

bool CheckHandOverHead(Vector3? headPosition, Vector3? handPosition)
{
    if (!handPosition.HasValue || !headPosition.HasValue)
        return false;

    if (handPosition.Value.Y < headPosition.Value.Y)
        return false;

    if (Math.Abs(handPosition.Value.X - headPosition.Value.X) > MaxRange)
        return false;

    if (Math.Abs(handPosition.Value.Z - headPosition.Value.Z) > MaxRange)
        return false;

    return true;
}

You will notice that this method checks to recognize a “hello” gesture by determining several different positions: