Inclusive musical educational game technology for children

Abstract

This research proposes a novel teaching aid aimed at music schools teaching music to students, including those with special needs. Previous research has successfully used modified or specifically built data gloves for this purpose. Data gloves worn by the teacher and the student were able to convey real time guidance and feedback by detecting the fingers’ state using strain gauges and conveying feedback to the student using miniature vibrating devices incorporated in the gloves. This study proposes a flexible approach where the gloves could either respond to live performance by a teacher or use existing MIDI music files as a source. This could allow teachers to cater for more students simultaneously and facilitate practising at home. For the purpose of this study, a set of fabric gloves were equipped with strain gauges which measure the angle of each finger, thereby detecting which notes are currently being pressed. Similarly, small vibrating devices were attached to the fingers in order to give feedback to the student or direct him/her to which are the correct notes to press. A third device equipped with a loudspeaker is also included to give auditory feedback to the user. The Espressif ESP32 was selected as the microcontroller platform for this project. The software was designed in C++ and developed using the Arduino IDE. Wi-Fi was chosen as the preferred communication medium due to its additional range and bandwidth over Bluetooth. This allows the various devices to communicate together as well as with user devices via a web interface which allows the users to control, monitor and upload MIDI files to the devices. A prototype device was built in order to evaluate the proposed technology, and case studies were held at a professional music school with a number of professional music teachers. The feedback unanimously confirmed that this technology could be helpful especially with beginners or students with special needs.