VR enhance : aiding human speech and sensorimotor skills using virtual reality

Abstract

Stroke remains one of the major causes for most language and functional disabilities, but this disease is not the only cause for such deficits. Current rehabilitation programs struggle to keep up with increasing demands for therapy, each day. The psychological and societal impacts of therapy must also not be underestimated, as such programs are usually very time consuming with large dependencies on the expertise of the therapist. This study presents VR-Enhance, a novel VR-based rehabilitation game system that uses multimodality to identify both speech and dynamic gestures within a single application. The solution aims to provide an alternate means of therapy by allowing patients to independently improve their speech and physical abilities, specifically those related to the upper extremities, with minimal guidance from therapists. For user engagement, the system applies themes of magic and spells to instantiate intra-diegetic features after speech or gesture classification, which are amplified based on the user’s score. A sensor-based deep neural network is applied, able of recognising both one-handed and two-handed gestures, essential for targeting bimanual activities. For speech, IBM Watson’s cloud-based speech-to-text service is used with streaming, to allow for continuous speech recognition until a pause is detected. The performance of both models is evaluated through a user evaluation to validate the efficacy of the proposed system. When applied to 18 participants, a global Accuracy and Cohen’s kappa of 93.3% and 89.9% respectively are achieved for the gesture model. These results indicate the model’s ability to extend to different users whilst maintaining considerable accuracies. An overall word error rate of 28.8% was achieved for the speech model, which suggests that further improvements are required to recognise speech with low intelligibility. Nonetheless, a gradual improvement in user scores was observed during the 10 repetitions performed for each gesture and speech sequence. The system was very well accepted by users, all giving an indication of possibly making use of VR for rehabilitation in the future.