Investigating the recognition of the static hand signs used by fingerspelling on a smart phone

Abstract

One of the leading challenges faced by the deaf members of society is the communication barrier which arises due to their lack of hearing and the lack of individuals who are able to express themselves through sign language. When considering both the deaf and the hearing communities, one factor common to both parties is the use of smart phone devices. However, whereas hearing individuals are able to freely communicate through audio calls on a smart phone, the latter is not the case for deaf individuals who rely on sign language as their principal means of communication. This thesis attempts to tackle this issue by correlating smart phone technologies with a subset of sign language communication. Specifically, it attempts to develop an automated system which recognises the various static hand postures which may be found within a sign language’s alphabet through the use of a smart phone device. This is done through an investigation of the existing technologies in the field of study, identification of possible challenges which may come up, as well as possible solutions on how to tackle them and finally the development of a smart phone application with the knowledge gained through this research. The main lifecycle for the recognition of these static hand postures involves passing an image through a series of image processing and feature extraction techniques, followed by a machine learning method that is used to recognize the static symbols captured by the camera. The recognition rate of the developed proof of concept is tested, which tops off at 83.7% average recognition. Through the tests performed, it was also concluded that this value may be further increased through the use of a larger training dataset and different extraction algorithms.