EEC-492/592
Kinect Application Development
Lecture 14
Wenbing Zhao
[email protected]
Outline

Unity3D + ZDK
Creating a New Unity Project


Open Unity, create a new project,
choose a location folder of your
choice
Important ZDK plugin



Go to Assets menu, select
“Import Package”, then “Custom
Package…”
In the dialogue, find and select
the ZDK plugin and click OK:
ZDK_Unity40_1.1_trial.unitypack
age
In the “Import package” dialogue,
keep default selection and click
Import button
Project Layout after ZDK is imported
Scene View

Where you plan and execute your ideas

To show the grid, toggle off the game overlay button
Hierarchy View



Shows what’s in the currently active scene.
GameObjects that are dynamically added and removed from
the scene during runtime will appear here when they are
active in the scene
By default, the main
camera is added
Constructing Scene

Add the following items to the scene:


Avatar: Dana (from ZDK)
 In Assets from Project panel, go to ZigFu folder, then
_Data folder
 Drag and drop Dana@t-pose_3 to the Hierarchy panel
 Rotation 180 degree so that it faces front
Floor

Add a Cube game object, change its size to a thin layer





Change material to floor (imported from ZDK)
Use Inspector, to go Mesh Renderer section, then materials
=> Element 0 => browse => choose Floor
Directional light
Empty object: to enable bootstrapping with ZDK



Scale: x=30, y=0.1, z=30
From GameObject menu, choose Create Empty
Rename it to InitZDK (right click the object, then Rename)
Adjust main camera so that Dana almost fill the entire
game view
Connecting Existing Scripts






Change Project panel to one column
Expand ZigFu folder, then expand Scripts folder and all subfolders
Locate Zig script, drag and drop it to InitZDK in the Hierarchy panel
Locate ZigDepthViewer script under the Viewers subfoler to InitZDK
Locate ZigUsersRadar script under the Viewers subfoler to InitZDK
Locate ZigEngageSingleUser script under the UserEngagers
subfoler to InitZDK



Drag and drop Dana from the Hierarchy panel to Engaged Users
section under Zig Engage Singler User in the inspector
Locate kinectSpecific script under the _Internal subfoler to InitZDK
Locate ZigSkeleton script under the UserControls subfoler to Dana
After Connecting Existing Scripts
Connecting Joints for Dana the Avatar


Expand the entire hierarchy of Dana mesh
Drag and drop matching joints from the Dana mesh in the hierarchy
panel to the appropriate joints under Zig Skeleton section in the inspector
 Head => Head
Left Hip => LeftUpLeg










Neck => Neck
Left Knee=> LeftLeg
Torso => Spine1
Left Ankle => LeftFoot
Waist => Spine
Right Hip => RightUpLeg
Left Shoulder => LeftArm
Right Knee => RightLeg
Left Elbow => LeftForeArm Right Ankle => RightFoot
Left Wrist => Left Hand
Right Shoulder => RightArm
Right Elbow => RightForeArm
Right Wrist => RightHand
Check the Mirror checkbox in the Dana inspector
Connecting Joints for Dana the Avatar
What Do the Scripts Do?

Zig.cs


Take user settings on what to update,
smoothing, and smoothing parameters
Public member variables can be set
via the inspector
public class Zig : MonoBehaviour {
public ZigInputType inputType = ZigInputType.Auto;
public ZigInputSettings settings = new
ZigInputSettings();
public List<GameObject> listeners = new
List<GameObject>();
public bool Verbose = true;
ZigDepthViewer.cs


OnGUI(): called for rendering and handling GUI events
static void DrawTexture(Rect position, Texture image,
ScaleMode scaleMode = ScaleMode.StretchToFill, bool
alphaBlend = true, float imageAspect = 0);





Position: Rectangle on the screen to draw the texture within.
Image: Texture to display.
scaleMode: How to scale the image when the aspect ratio of it
doesn't fit the aspect ratio to be drawn within.
alphaBlend: Whether to apply alpha blending when drawing the
image (enabled by default).
imageAspect: Aspect ratio to use for the source image
void OnGUI() {
if (null == target) {
GUI.DrawTexture(new Rect(Screen.width - texture.width - 10,
Screen.height - texture.height - 10, texture.width, texture.height), texture);
}
}
ZigDepthViewer.cs


Texture: the visual and especially tactile quality of a
surface. Parent class for Texture2D
Texture2D reference


http://docs.unity3d.com/Documentation/ScriptReference/Tex
ture2D.html
You can also add a button
void OnGUI() {
if (GUI.Button(new Rect(10, 10, 150, 100), "I am a button"))
print("You clicked the button!");
}
ZigDepthViewer.cs

Getting input from Kinect
void Zig_Update(ZigInput input) {
if (UseHistogram) {
UpdateHistogram(ZigInput.Depth); // ZigInput.Depth contains the depth data
} else {
depthToColor[0] = Color.black;
for (int i = 1; i < MaxDepth; i++) {
float intensity = 1.0f - (i/(float)MaxDepth);
depthToColor[i].r = (byte)(BaseColor.r * intensity);
depthToColor[i].g = (byte)(BaseColor.g * intensity);
depthToColor[i].b = (byte)(BaseColor.b * intensity);
depthToColor[i].a = 255;
}
}
UpdateTexture(ZigInput.Depth);
}
ZigDepthViewer.cs

Update Depth Image
void UpdateTexture(ZigDepth depth)
{
short[] rawDepthMap = depth.data;
int depthIndex = 0;
int factorX = depth.xres / textureSize.Width;
int factorY = ((depth.yres / textureSize.Height) - 1) * depth.xres;
// invert Y axis while doing the update
for (int y = textureSize.Height - 1; y >= 0 ; --y, depthIndex += factorY) {
int outputIndex = y * textureSize.Width;
for (int x = 0; x < textureSize.Width; ++x, depthIndex += factorX, ++outputIndex) {
outputPixels[outputIndex] = depthToColor[rawDepthMap[depthIndex]];
}
}
texture.SetPixels32(outputPixels);
texture.Apply();
}
ZigUserRadar.cs: Track where the user is
void OnGUI () {
if (!ZigInput.Instance.ReaderInited) return;
int width = (int)((float)PixelsPerMeter * (RadarRealWorldDimensions.x / 1000.0f));
int height = (int)((float)PixelsPerMeter * (RadarRealWorldDimensions.y / 1000.0f));
GUI.BeginGroup (new Rect (Screen.width - width - 20, 20, width, height));
Color oldColor = GUI.color; GUI.color = boxColor;
GUI.Box(new Rect(0, 0, width, height), "Users Radar", style); GUI.color = oldColor;
foreach (ZigTrackedUser currentUser in ZigInput.Instance.TrackedUsers.Values) {
// normalize the center of mass to radar dimensions
Vector3 com = currentUser.Position;
Vector2 radarPosition = new Vector2(com.x / RadarRealWorldDimensions.x, -com.z /
RadarRealWorldDimensions.y);
// X axis: 0 in real world is actually 0.5 in radar units (middle of field of view)
radarPosition.x += 0.5f;
radarPosition.x = Mathf.Clamp(radarPosition.x, 0.0f, 1.0f);
radarPosition.y = Mathf.Clamp(radarPosition.y, 0.0f, 1.0f);
Color orig = GUI.color;
GUI.color = (currentUser.SkeletonTracked) ? Color.blue : Color.red;
GUI.Box(new Rect(radarPosition.x * width - 10, radarPosition.y * height - 10, 20, 20),
currentUser.Id.ToString());
GUI.color = orig;
}
GUI.EndGroup();
}
ZigEngageSingleUser.cs

Connect ZigInput to the avatar: that is why you must drag and
drop Dana to EngagedUsers field
public class ZigEngageSingleUser : MonoBehaviour {
public bool SkeletonTracked = true;
public bool RaiseHand;
public List<GameObject> EngagedUsers;
void Start() {
// make sure we get zig events
ZigInput.Instance.AddListener(gameObject);
}
void Zig_Update(ZigInput zig) {
if (SkeletonTracked && null == engagedTrackedUser) {
foreach (ZigTrackedUser trackedUser in zig.TrackedUsers.Values) {
if (trackedUser.SkeletonTracked) {
EngageUser(trackedUser);
}
}
}
}
ZigEngageSingleUser.cs
void EngageUser(ZigTrackedUser user) {
if (null == engagedTrackedUser) {
engagedTrackedUser = user;
foreach (GameObject go in EngagedUsers) user.AddListener(go);
SendMessage("UserEngaged", this, SendMessageOptions.DontRequireReceiver);
}
}
Component.SendMessage:
void SendMessage(string methodName, object value = null,
SendMessageOptions options = SendMessageOptions.RequireReceiver);
// Calls the method named methodName on every MonoBehaviour in this game object
kinectSpecific.cs: Kinect specific settings
void OnGUI() {
longWord = GUI.TextField(new Rect(10, 10, 200, 30), readingAngle ? getAngle().ToString() : longWord, 20);
if (GUI.Button(new Rect(10, 40, 200, 30), "SetElevation")) {
angle = int.Parse(longWord);
NuiWrapper.NuiCameraElevationSetAngle(angle);
t = new Thread(setAngle); //attempted a Paramaterized Thread to no avail
t.Start(); Thread.Sleep(0);
}
readingAngle = GUI.Toggle(new Rect(10, 80, 200, 30), readingAngle, "Read Angle");
bool nNearMode = GUI.Toggle(new Rect(10, 160, 200, 20), NearMode, "Near Mode");
if (nNearMode != NearMode) {
NearMode = nNearMode;
ZigInput.Instance.SetNearMode(NearMode);
}
bool nSeatedMode = GUI.Toggle(new Rect(10, 190, 200, 20), SeatedMode, "Seated Mode");
bool nTrackSkeletonInNearMode = GUI.Toggle(new Rect(10, 220, 200, 20),
TrackSkeletonInNearMode, "Track Skeleton In NearMode");
if ((nSeatedMode != SeatedMode) || (TrackSkeletonInNearMode != nTrackSkeletonInNearMode)) {
SeatedMode = nSeatedMode;
TrackSkeletonInNearMode = nTrackSkeletonInNearMode;
ZigInput.Instance.SetSkeletonTrackingSettings(SeatedMode, TrackSkeletonInNearMode);
}
}
ZigSkeleton.cs
public class ZigSkeleton : MonoBehaviour
{
public Transform Head; public Transform Neck; public Transform Torso; public Transform Waist;
public Transform LeftCollar; public Transform LeftShoulder; public Transform LeftElbow;
public Transform LeftWrist; public Transform LeftHand; public Transform LeftFingertip;
public Transform RightCollar; public Transform RightShoulder; public Transform RightElbow;
public Transform RightWrist; public Transform RightHand; public Transform RightFingertip;
public Transform LeftHip; public Transform LeftKnee; public Transform LeftAnkle;
public Transform LeftFoot;
public Transform RightHip; public Transform RightKnee; public Transform RightAnkle;
public Transform RightFoot;
public bool mirror = false;
public bool UpdateJointPositions = false; public bool UpdateRootPosition = false;
public bool UpdateOrientation = true; public bool RotateToPsiPose = false;
public float RotationDamping = 30.0f; public float Damping = 30.0f;
public Vector3 Scale = new Vector3(0.001f, 0.001f, 0.001f);
public Vector3 PositionBias = Vector3.zero;
private Transform[] transforms; private Quaternion[] initialRotations; private Vector3 rootPosition;
Quaternions are used to represent rotations, they are based on complex numbers and are
not easy to understand intuitively
ZigSkeleton.cs
ZigJointId mirrorJoint(ZigJointId joint) {
switch (joint) {
case ZigJointId.LeftCollar:
return ZigJointId.RightCollar;
case ZigJointId.LeftShoulder:
return ZigJointId.RightShoulder;
case ZigJointId.LeftElbow:
return ZigJointId.RightElbow;
case ZigJointId.LeftWrist:
return ZigJointId.RightWrist;
case ZigJointId.LeftHand:
return ZigJointId.RightHand;
case ZigJointId.LeftFingertip:
return ZigJointId.RightFingertip;
case ZigJointId.LeftHip:
return ZigJointId.RightHip;
…..
// map right to left
default:
return joint;
}
}
ZigSkeleton.cs
public void Awake() {
int jointCount = Enum.GetNames(typeof(ZigJointId)).Length;
transforms = new Transform[jointCount];
initialRotations = new Quaternion[jointCount];
transforms[(int)ZigJointId.Head] = Head;
transforms[(int)ZigJointId.Neck] = Neck;
transforms[(int)ZigJointId.Torso] = Torso;
transforms[(int)ZigJointId.Waist] = Waist;
…..
// save all initial rotations
// NOTE: Assumes skeleton model is in "T" pose since all rotations are relative to that pose
foreach (ZigJointId j in Enum.GetValues(typeof(ZigJointId)))
{
if (transforms[(int)j])
{
// we will store the relative rotation of each joint from the gameobject rotation
// we need this since we will be setting the joint's rotation (not localRotation) but we
// still want the rotations to be relative to our game object
initialRotations[(int)j] = Quaternion.Inverse(transform.rotation) * transforms[(int)j].rotation;
}
}
}
ZigSkeleton.cs
void Zig_UpdateUser(ZigTrackedUser user) {
UpdateRoot(user.Position);
if (user.SkeletonTracked) {
foreach (ZigInputJoint joint in user.Skeleton)
{
if (joint.GoodPosition) UpdatePosition(joint.Id, joint.Position);
if (joint.GoodRotation) UpdateRotation(joint.Id, joint.Rotation);
}
}
}
void UpdateRoot(Vector3 skelRoot) {
// +Z is backwards in OpenNI coordinates, so reverse it
rootPosition = Vector3.Scale(new Vector3(skelRoot.x, skelRoot.y, skelRoot.z), doMirror(Scale))
+ PositionBias;
if (UpdateRootPosition)
{
transform.localPosition = (transform.rotation * rootPosition);
}
}
Vector3.Scale(Vector3 a, Vector3 b): Multiplies two vectors component-wise.
ZigSkeleton.cs
void UpdateRotation(ZigJointId joint, Quaternion orientation) {
joint = mirror ? mirrorJoint(joint) : joint;
// make sure something is hooked up to this joint
if (!transforms[(int)joint]) { return; }
if (UpdateOrientation) {
Quaternion newRotation = transform.rotation * orientation * initialRotations[(int)joint];
if (mirror) { newRotation.y = -newRotation.y; newRotation.z = -newRotation.z; }
transforms[(int)joint].rotation = Quaternion.Slerp(transforms[(int)joint].rotation,
newRotation, Time.deltaTime * RotationDamping);
}
static Quaternion Slerp(Quaternion from, Quaternion to, float t);
}
// Spherically interpolates between from and to by t.
void UpdatePosition(ZigJointId joint, Vector3 position) {
joint = mirror ? mirrorJoint(joint) : joint;
if (!transforms[(int)joint]) { return; }
if (UpdateJointPositions) {
Vector3 dest = Vector3.Scale(position, doMirror(Scale)) - rootPosition;
//Vector3.Lerp: Linearly interpolates between two vectors.
transforms[(int)joint].localPosition = Vector3.Lerp(transforms[(int)joint].localPosition,
dest, Time.deltaTime * Damping);
}
}