by Martin Fröhlich
HeadRoom, the missing kinect network solution for interaction space developers.
It is based on openFrameworks and libfreenect, should run on OSX (tested), Windows and Linux.
NOTE: Xbox Kinect models > 1414 (1473, etc) or Kinect4Windows devices that have been plugged into an XBox will not work with ofxKinect on Mac OSX
If you have an Xbox Kinect model 1473+ or Kinect4Windows device and want to use it with ofxKinect on OSX, DO NOT plug it into an XBox! If you do, it will receive a firmware update which will cause it to freeze after about 20 secs with libfreenect/ofxKinect on OSX. Linux and Windows are fine. The fix is in the works.
In the meantime, we suggest you get the original Xbox Kinect model 1414 and keep those new toys away from an XBox ...
HeadRoom runs as an application on a dedicated computer with one attached kinect device. The machine needs to be powerfull enough to do all the tracking analysis necessary.
It is built for a accurate tracking of bodies. The tracker can send the bodies position and heigth, head-toptip-position, head-center-position, eye-center-position and gaze direction.
It has a simple but powerfull calibration functionality. This makes it very easy to tell how the kinect is located towards a floor, ceilling or a wall.
All data is in a metric cartesian coordinatesystem referenced from a floor.
The results are sent as UDP and/or TCP data over the network to a requesting client.
On startup, HeadRoom will send every second a OSC-broadcast-announcement to the broadcast address (of the newtork the machine resides in) to port 43500. The server itself will listen to a different port (default = 43600), indicated inside the broadcast message.
Assuming the IP-address of the kinectServer is 192.168.1.100, it will send the handshake to 192.168.1.255 / 43500.
/ks/server/broadcast <kinectid> <serverID> <ServerIP> <ServerListeningPort>
for example
/ks/server/broadcast A00363A14660053A 0 192.168.1.100 43600
every client in the network can respond to this broadcast and send a handshake request back to the HeadRoom:
/ks/request/handshake <ClientListeningPort>
upon receiving this request, the server will send the calibration data. If the client misses some of the calibration data it has to resend the handshake request.
In order to get the trackingdata, the server then needs an update message every 10 seconds.
/ks/request/update <ClientListeningPort>
upon receiving this request, the server will send a continous stream of the tracking data for the next 11 seconds. Since the server will keep on sending its broadcast message every 10 seconds, the clients resend of the update-message can be triggered by the broadcast-message and thus the connection will never drop.
HeadRoom will stop sending the stream of tracking data if no update-message is received anymore and drop the registration of the client.
If no client is registered anymore, the server will again start sending the broadcast message every second.
If the server encounters a problem or stops, it will atempt to broadcast an exit message:
/ks/server/broadcast/exit <kinectid> <serverID>
##calibration data
####transformation
/ks/server/calib/trans <serverID> <x-rotate[deg]> <y-rotate[deg]> <z-translate[m]>
with this info the kinect transformation matrix in relation to the floor can be calcualted like this (example code for openframeworks):
kinectTransform = ofMatrix4x4();
kinectTransform.rotate(<x-rotate>, 1, 0, 0);
kinectTransform.rotate(<y-rotate>, 0, 1, 0);
kinectTransform.translate(0, 0, <z-translate>);
The transformation matrix is mainly used to correctly transform the frustum and pointcloud data.
####kinects frustum:
/ks/server/calib/frustum <serverID> <left[m]> <right[m]> <bottom[m]> <top[m]> <near[m]> <far[m]>
the frustum needs to translated by the above transformation matrix to be in the correct space
####sensorbox:
/ks/server/calib/sensorbox <serverID> <left(x-axis)[m]> <right(x-axis)[m]> <bottom(z-axis)[m]> <top(z-axis)[m]> <near(y-axis)[m]> <far(y-axis)[m]>
####gaze:
/ks/server/calib/gazepoint <serverID> <gazePosX[m]> <gazePosY[m]> <gazePosZ[m]>
##tracking data
####frame
/ks/server/track/frame <serverID> <frameNo> <sendBodyBlob> <sendHeadBlob> <sendHead> <sendEye>
the message is sent each time at the beginning of a new frame. 'sendBodyBlob', 'sendHeadBlob', 'sendHead', 'sendEye' are 1 if beeing sent.
####bodyBlob
/ks/server/track/bodyblob <serverID> <frameNo> <blobID> <sortPos> <bodyBlobPosX[m]> <bodyBlobPosY[m]> <bodyBlobWidth(x-axis)[m]> <bodyBlobDepth(y-axis)[m]> <bodyHeight(z-axis)[m]>
####headBlob
/ks/server/track/headblob <serverID> <frameNo> <blobID> <sortPos> <headBlobPosX[m]> <headBlobPosY[m]> <headBlobPosZ[m]> <headBlobWidth(x-axis)[m]> <headBlobDepth(y-axis)[m]>
####head
/ks/server/track/head <serverID> <frameNo> <blobID> <sortPos> <headTopPosX[m]> <headTopPosY[m]> <headTopPosZ[m]> <headCenterPosX[m] <headCenterPosY[m] <headCenterPosZ[m]>
####eye
/ks/server/track/eye <serverID> <frameNo> <blobID> <sortPos> <eyePosX[m]> <eyePosY[m]> <eyePosZ[m]> <eyeGazeX> <eyeGazeY> <eyeGazeZ>
eyeGazeX, Y, Z is a normalized vector. Beware: The gaze is calculated based on a defined gaze-point and not through facial feature tracking. It assumes that each tracked person looks at this gaze-point.
####framedone
/ks/server/track/framedone <serverID> <frameNo>
the last message sent for the current frame.
##Download and installation
in order to compile this app on your system, you need openframeworks.
###openFrameworks You need to install the current master branch from github, check the download page on the main site.
Check out the Commit fc829d5 from 05/10/2017 by arturo. Some of the later commits are missing the libs/openFrameworks/types/ofBaseTypes.h file, which is required by the ofxGuiExtended external. (see below)
###HeadRoom To grab a copy of HeadRoom for your platform, check here download page.
The master branch of this repository corresponds to the most recent release. This GitHub repository contains code and libs for all the platforms.
drop it into the <openframeworksfolder>/apps/<yourappfolder>
you also need a copy of the openframeworks addon ofxGuiExtended
drop it onto the <openframeworksfolder>/addons/<ofxGuiExtended>
##How to to come...
##Version HeadRoom uses Semantic Versioning,
Version 0.0.2
- switch to ofxGuiExtended
- redesign of GUI structure
- removal of MatrixServer
Version 0.0.1
- initial release
##Credits
(c) by tecartlab.com
created by Martin Froehlich for tecartlab.com
loosely based on a concept by Andrew Sempre and his performance-space
##Licence MIT and see license.md