Microsoft Kinect for Java and Eclipse in Jnect
An area where we find a high degree of innovation is in the way we communicate with computers and other electronic devices. Touch displays are almost a must-have for mobiles and speech control could be the next big thing in this area. Looking at the games market, every major player has a device allowing users to control games and even media features, using gestures instead of gamepads.
Microsoft Kinect is an example of such a device. Its cameras enable the tracking of a moving human body and provide this information to the Xbox 360 to interact with the game. For example, if the player jumps in the real world, its avatar in the game will also perform a jump. The precise tracking of Kinect also enables the control of more serious features, for example the selection of a movie to be watched. Additionally, Kinect is able to understand spoken commands using speech recognition.
Figure 1: All positions of a body tracked by Kinect can be visualized in real time.
Obviously, this technology could be used for more than controlling the Xbox. Microsoft has published an SDK for Windows, providing programmatic access to the data provided by the Kinect device. This opens the potential use cases for Kinect to almost any kind of application.
However, the SDK requires the programmer to write the application in C++/C# to access the SDK, and to date, there has been no support for Java. Jnect bridges this gap and creates interoperability between the Microsoft SDK and the Java world. The project provides an Eclipse plugin which uses JNI to connect with the Microsoft Kinect SDK. The plugin provides a Java API to use selected features offered by the original Microsoft SDK (see below).
Figure 2: Jnect connects to the Microsoft Kinect SDK and provides a Java API for selected features
The body tracking feature provides access to the positions of various parts of a human body being tracked by the Kinect camera. Through the Jnect API, you can create applications that exploit the body tracking and speech abilities of the Kinect using Java and Eclipse RCP. For example, in Java, the following simple statement retrieves the x-position of the left hand:
The positions of the various parts of the body are updated in real-time as the person moves. Figure 1 shows a body being tracked by Kinect, drawn on a GEF diagram in Eclipse. See for a video here showing this example in action.
The precise information about positions of single parts of the body also provides the data required for gesture recognition. With the following snippet, we can compare the x-position of the right hand with the position of the head to indicate that a person has raised his hand.
float head = kinectManager.getSkeletonModel().getHead().getY(); float rHand = kinectManager.getSkeletonModel().getRightHand().getY(); return y<y2;
Finally, Jnect also enables the speech recognition feature of the Microsoft Kinect SDK. With this feature we define a set of phrases the Kinect recognizes, which in turn trigger a particular action, such as 'open a new window'.
Jnect, created in collaboration with a group of interested students, has published an open source beta version of the framework, as an Eclipse Labs project. The website offers a collection of demonstrations and code examples. One example you can try out is to resize the Eclipse debugger by moving two hands. The Jnect team hopes you'll be inspired by the initial contribution and welcomes your feedback and contributions!