EEC-492/592
Kinect Application Development
Lecture 10
Wenbing Zhao
[email protected]
Outline

Human skeleton tracking (part III)
Skeleton Smoothing

Skeleton data can be smoothed by registering a set of
smooth parameters

As a parameter to SkeletonStream.Enable(….);
// create the smooth parameters
var smoothParameters = new TransformSmoothParameters
{
Correction = 0.1f,
JitterRadius = 0.05f,
MaxDeviationRadius = 0.05f,
Prediction = 0.1f,
Smoothing = 1.0f
};
// Enable the skeleton stream with smooth parameters
this.sensor.SkeletonStream.Enable(smoothParameters);
Smoothing Parameters

Correction: specifies the amount of correction
needed for the raw data. Range [0.0, 1.0]



The value must be within the range of 0 to 1.0 and the
default value is 0.5.
With lower values more correction is applied, the raw data is
corrected, and the data looks smoother
Smoothing: determines the amount of smoothing
applied while processing. Range [0.0, 1.0]



The value must be within the range of 0 to 1.0 and the
default value is 0.5.
The larger value, the smoother skeleton data, however, it
increases the latency
Zero value => you will get the raw skeleton data
Smoothing Parameters

JitterRadius: limit the radius value for jittery data, >=0



MaxDeviationRadius: max limit of the deviation that is
allowed to be considered for determining a jitter, >=0



Measured in meters and the default value is 0.5.
If the position of a jitter is outside the set radius, it is
corrected to be positioned at the radius
points outside of the MaxDeviationRadius range are not
considered as jitter
default value is 0.04 meters
Prediction: number of frames predicted into the future, >=0


default value: 0.0
A value greater than 0.5 will likely lead to overshoot when
the data changes quickly
Kinect SDK Smoothing Algorithm

Holt double exponential smoothing algorithm is
used to reduce the jitters from skeletal joint data


The smoothing algorithm applies to each set of data and
calculates a moving average based on the previous set of
data
During the calculation of moving average, it uses the values
passed by the smoothing parameter.
Getting Data Frames Together




We have covered three (color, depth, skeleton) types of data
streams that are returned by the sensor
For a real application, we often need all three types of data
streams
We can use a single event AllFramesReady, which will do the
job for all three of them
The AllFramesReady event fires when new frames are available
for the color, depth, and skeleton streams
this.sensor.AllFramesReady+=sensor_AllFramesReady;
void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
}
Getting Data Frames Together
Floor Determination

Each skeleton frame contains a floor-clipping-plane
vector, which contains the coefficients of an
estimated floor-plane equation

The equation is normalized so that the physical
interpretation of D is the height of the camera from the
floor, in meters
Ax + By + Cz + D = 0

The skeleton tracking system updates this estimate
for each frame and uses it as a clipping plane for
removing the background and segmenting players
How to Use FloorClipPlane In Your App



FloorClipPlane can be used to calculate the height
of each joint with respect to the floor
Can be used for validation study, as well as fall
detection
How to get FloorClipPlane parameters:
float A = frame.FloorClipPlane.Item1;
float B = frame.FloorClipPlane.Item2;
float C = frame.FloorClipPlane.Item3;
float D = frame.FloorClipPlane.Item4;

How to calculate the height of a joint:
float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;
Joint Orientation

The bone orientation is provided in two forms:




A hierarchical rotation based on a bone relationship defined on
the skeleton joint structure
An absolute orientation in Kinect camera coordinates
The orientation information is provided in form of quaternions
and rotation matrices for use in different animation scenarios
BoneOrientation Class, public properties:




StartJoint: Gets the skeleton joint where the bone starts
EndJoint: Gets the skeleton joint where the bone ends
HierarchicalRotation: Gets or sets the rotation of a bone relative to its
parent bone, of type BoneRotation
AbsoluteRotation: Gets or sets the rotation of the bone relative to
camera coordinates, of type BoneRotation
BoneRotation Class

BoneRotation has two properties


Matrix: Gets or sets a matrix representation of the bone rotation
Quaternion: Gets or sets a quaternion representation of the bone
rotation
public Matrix4 Matrix { get; set; }
public Vector4 Quaternion { get; set; }

Vector4 struct: has four properties


W, X, Y, Z
Matrix4 struct: has 17 properties


Identity, of type Matrix4
M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33,
M34, M41, M42, M43, M44
Access Joint Orientation
private void DrawSkeletonsWithOrientations()
{
foreach (Skeleton skeleton in this.skeletonData) {
if (skeleton.TrackingState == SkeletonTrackingState.Tracked) {
foreach (BoneOrientation orientation in
skeleton.BoneOrientations)
{ // Display bone with Rotation using quaternion
DrawBonewithRotation(orientation.StartJoint,
orientation.EndJoint,
orientation.AbsoluteRotation.Quaternion);
// Display hierarchical rotation using matrix
DrawHierarchicalRotation(orientation.StartJoint,
orientation.HierarchicalRotation.Matrix)
}
}
}
}
Build ShapeGame App

Create a new C# WPF project with name ShapeGame








It is a much simplified app from the Kinect ShapeGame
Only one shape, i.e., a ball is used
You are limited to use your righthand to hit the ball
Ball only drops from a designated spot down, i.e., no X velocity
Add Microsoft.Kinect reference
Design GUI
Added WindowLoaded() method in xaml file
Adding code
GUI Design
Image control
Canvas
Adding Code

Add member variables:
KinectSensor sensor;
Skeleton[] totalSkeleton = new Skeleton[6];
WriteableBitmap colorBitmap;
byte[] colorPixels;
Skeleton skeleton;
Thing thing = new Thing(); // a struct for ball
double gravity = 0.017;

WindowLoaded():



Enable both ColorImageStream and SkeletonStream
Register event handler for both ColorFrameReady and
SkeletonFrameReady events
Initialize the shape (i.e., ball)
Adding Code

The Thing Struct. Make it a private struct inside the MainWindow class
private struct Thing
{
public System.Windows.Point Center;
public double YVelocity;
public double XVelocity;
public Ellipse Shape;
public bool Hit(System.Windows.Point joint)
{
double minDxSquared = this.Shape.RenderSize.Width;
minDxSquared *= minDxSquared;
double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y);
if (dist<= minDxSquared)
{
return true;
} else
return false;
}
}
Adding Code
private static double SquaredDistance(double x1, double y1, double x2, double y2)
{
return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1));
}
private void WindowLoaded(object sender, RoutedEventArgs e)
{
// same as before
….
// new code for ball initialization
thing.Shape = new Ellipse();
thing.Shape.Width = 30;
thing.Shape.Height = 30;
thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255));
thing.Center.X = 300; thing.Center.Y = 0;
thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width);
thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width);
canvas1.Children.Add(thing.Shape);
}
Modify Event Handler for Skeleton
Frames
void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
canvas1.Children.Clear();
advanceThingPosition();
canvas1.Children.Add(thing.Shape);
// remaining code same as before
…..
Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position);
if(thing.Hit(handPt))
{
this.thing.YVelocity = -1.0*this.thing.YVelocity;
}
}
Shape Animation Code
void advanceThingPosition()
{
thing.Center.Offset(thing.XVelocity, thing.YVelocity);
thing.YVelocity += this.gravity;
thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width);
thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width);
// if goes out of bound, reset position, as well as velocity
if (thing.Center.Y >= canvas1.Height)
{
thing.Center.Y = 0;
thing.XVelocity = 0;
thing.YVelocity = 0;
}
}
Challenge Task

For advanced students, improve the shape game in
the following ways:





7/31/2017
Keep the hit count, count should reset if ball falls to the
bottom
Display the hit count at the ball
Consider the angle of the hit to the ball, and adjust the x
velocity accordingly
Allow other joints to hit the ball too, such as left hand,
shoulder, etc.
Add smoothing to skeleton data
EEC492/693/793 - iPhone Application Development
21