This app applies Open Pose to simultaneously track the movements of two players, from two separate video feeds. The players can physically interact in a virtual space and play a boxing game, even if they are in separate rooms! This was created for the ImplementAI 2019 Hackathon.
This method is really flexible and can be run on a standard laptop with no special equipment. We further demonstrated the flexibility by creating two-player physical versions of Atari and Pong, where each player controls their moves with physical movements instead of traditional key strokes. This brings back retro games with a new interactive twist that transforms sedentary individual games into collaborative exercise.
Demo: Stand-alone web Atari Breakout using posenet for controls
Demo: Stand-alone web multiplayer Pong using posenet for controls
- Local clients uses Flask to broadcast each player's video feeds on the local area network via a multi-threaded video capture class.
- Use ngrok port forwarding to make each video feed public accessible
- Pose estimation server hosted on Google Colab accesses the video feeds
- Server estimates joint landmark vector for each player via a modified Open Pose model.
- Server streams back joint vectors via Flask and additional ngrok tunneling
- Local client processes joint vectors and reconstruct skeletal representation of each player in a common virtual space
- Further movement analysis and interaction leads to abilities such as virtual boxing matches and pose mimcking
- online deployment that allows users to connect via a website and removes need for local client
- upgrade from Open Pose to wrnch API for more accurate tracking (especially occluded joints) for estimating more complex player dynamics
- upgrade graphics to Unity Game Engine (3D characters, realistic textures etc)
- Python 3+
- OpenCV
- Open Pose
- Tensorflow
- ngrok
- Google Colab
- Flask