Enhancing stock price prediction models by using concept drift detectors

Abstract

Due to recent advances made in the field of machine learning, various research has been done on the issue of applying machine learning models to the stock market. As stated by the efficient market hypothesis, the market is constantly fluctuating and due to it’s dynamic nature, certain underlying concepts start to change over time. This phenomena is known as concept drift. When concept drift occurs the performance of machine learning models tends to suffer, sometimes drastically. This decline in performance occurs because the data distributions that were used to train the model are no longer in-line with the current data distribution. This dissertation contributes four retraining processes to help mitigate the problem of concept drift. In this dissertation a state-of-the-art Adv-ALSTM model is used together with a HDDMA concept drift detector. Every time the HDDMA concept drift detector detects a concept drift, the model undergoes one of the four possible retraining methods. In the evaluation, the results of the vanilla model are compared to the results of the models that are fitted with a concept drift detector. The conducted experiments highlight the effectiveness of each of the proposed retraining methods, as well as how each of the methods mitigates the negative effects of concept drift in different ways. The best observed results were a 2.5% increase in accuracy and a 135.38% increase in MCC when compared to the vanilla model. These results validate the effectiveness of the proposed retraining methods, and highlight how important it is for a machine learning model to address concept drift.