Motion carried

Microsoft Kinect for Java and Eclipse in Jnect

JonasHelming
Kinect

Jonas Helming tells us more about an exciting new Eclipse Labs project, linking up the motion-tracking capabilities of Microsoft Kinect with Eclipse and Java, through the use of the device’s SDK.

An area where we find a high degree of innovation is in the way
we communicate with computers and other electronic devices. Touch
displays are almost a must-have for mobiles and speech control
could be the next big thing in this area. Looking at the games
market, every major player has a device allowing users to control
games and even media features, using gestures instead of
gamepads.

Microsoft Kinect is an example of such a device. Its cameras
enable the tracking of a moving human body and provide this
information to the Xbox 360 to interact with the game. For example,
if the player jumps in the real world, its avatar in the game will
also perform a jump. The precise tracking of Kinect also enables
the control of more serious features, for example the selection of
a movie to be watched. Additionally, Kinect is able to understand
spoken commands using speech recognition.

Figure
1: All positions of a body tracked by Kinect can be visualized in
real time.

Obviously, this technology could be
used for more than controlling the Xbox. Microsoft has published an
SDK for Windows, providing programmatic access to the data provided
by the Kinect device. This opens the potential use cases for Kinect
to almost any kind of application.

However, the SDK requires the
programmer to write the application in C++/C# to access the SDK,
and to date, there has been no support for Java. Jnect bridges this
gap and creates interoperability between the Microsoft SDK and the
Java world. The project provides an Eclipse plugin which uses JNI
to connect with the Microsoft Kinect SDK. The plugin provides a
Java API to use selected features offered by the original Microsoft
SDK (see below).

Figure
2: Jnect connects to the Microsoft Kinect SDK and provides a
Java API for selected features

The body tracking feature provides
access to the positions of various parts of a human body being
tracked by the Kinect camera. Through the Jnect API, you can create
applications that exploit the body tracking and speech abilities of
the Kinect using Java and Eclipse RCP. For example, in Java, the
following simple statement retrieves the x-position of the left
hand:

kinectManager.getSkeletonModel().getLeftHand().getX();

The positions of the various parts
of the body are updated in real-time as the person moves. Figure 1
shows a body being tracked by Kinect, drawn on a GEF diagram in
Eclipse.
See for a video here showing this example in action
.

The precise information about
positions of single parts of the body also provides the data
required for gesture recognition. With the following snippet, we
can compare the x-position of the right hand with the position of
the head to indicate that a person has raised his hand.

float head = kinectManager.getSkeletonModel().getHead().getY();
float rHand = kinectManager.getSkeletonModel().getRightHand().getY();
return y<y2;

Finally, Jnect also enables the
speech recognition feature of the Microsoft Kinect SDK. With this
feature we define a set of phrases the Kinect recognizes, which in
turn trigger a particular action, such as ‘open a new window’.

Jnect, created in collaboration
with a group of interested students, has published an open source
beta version of the framework, as an Eclipse Labs
project
. The website offers a collection of demonstrations and

code examples
. One example you can try out is to resize the
Eclipse debugger by moving two hands. The Jnect team hopes you’ll
be inspired by the initial contribution and welcomes your feedback
and contributions!

Author
Comments
comments powered by Disqus