Hand gesture recognition for an augmented reality sandbox

Abstract

Children are always expressing their creativity and developing their thinking skills through various methods, although to be able to enhance this creativity in an educational manner would be highly beneficial. To link creativity with education, an interactive sandbox can be an effective tool, whereby the children using their hands and playing in sand, they can learn important life skills such as drawing letters or numbers, whilst remaining engaged at the same time. The aim of this project is to design and implement an algorithm which is able to perform hand detection, motion tracking and gesture recognition of the user’s hand in a sandbox region using a Microsoft Kinect for Windows RGB-D camera. This was achieved through familiarisation with similar works for inspiration and using various algorithms to arrive at a similar goal. The algorithm consists of a unique method which consists of four main parts. The first part detects the hand using depth imaging techniques and extracts information about the hand and is performed only once. This information is then utilized to strengthen the rest of the algorithm. The motion tracking uses RGB colour thresholded imaging to perform Normalized Cross-correlation hand detection and the Kalman filter for motion tracking. The algorithm then swaps again to depth imaging for Boundary Signature based gesture recognition. This algorithm was then tested across all of its sections and the results of these tests were discussed. From the results obtained it can be concluded that the objectives were reached. The algorithm designed could successfully track a user’s hand motion and read the gestures performed with a mean accuracy of 95.9% at a latency of 0.2868s. Thus, successfully tracking the gestures and motion of the hand with a low enough latency to provide a smooth feel to the algorithm.