Automatic scoring of local English essays using deep learning techniques

Abstract

Traditionally, scoring texts written by students was manually performed by human graders and considered to be a laborious and time-consuming task. In order to overcome this problem, automated essay scoring was introduced and its promising results prompted researchers to implement and enhance systems which could improve the traditional approach. Over the years, feature-engineered and deep learning approaches, combined with a series of Natural Language Processing techniques have been adopted within the field of automated assessment. Even though several automated text scoring systems exist and are applicable to English tests, there is still absence of these resources within the Maltese education. Currently, Maltese educators and students have to rely on manual grading for essays written in both the English and Maltese language. This work investigates the existing automated scoring systems with the aim of introducing a system that can be applied by Maltese educators to score essays written during assignments and examinations. We perform a number of empirical experiments to find the optimal model that can be used for scoring essays. These are based on neural models presented by other researchers in the literature reviewed. We adopt deep learning technologies throughout, and also extract several linguistic features to provide further information on the essay texts. The initial experiments are held on an English dataset which has been applied to a number of neural models in the Automated Essay Scoring (AES) field. The models obtained from these experiments are then used with a dataset composed of English essays written by Maltese students in examinations held locally. The Quadratic Weighted Kappa (QWK) metric is used to evaluate the models obtained. The best models are determined by considering the 4 essay types tackled in this research. An average QWK of 0.729 was obtained overall, which is lower than results reported in two of our baseline models. The main contribution of our research is towards scoring locally-written English essays. Although the highest result obtained from these experiments was that of 0.109, the experiments conducted provide a solid baseline research that exposes the difficulties present and the possible future work that can be performed to bring AES a step closer to being adopted within the Maltese education sector.