Investigating training set and learning model selection for churn prediction in online gaming

Abstract

Customer retention has always been a key performance indicator for businesses, as it has a direct impact on profit and revenue. In order for a business to control customer retention, the customer churn rate needs to be controlled. Customer churn, is the action of a customer dissociating itself from the business, hence directly affecting customer retention. Machine Learning algorithms are gaining momentum in order to control customer retention and churn. Furthermore, data related to customer churn possesses changing trends and variability (denoted by concept drift), which decreases the machine learning model predictive power. This decrease in predictive power is noted when predicting distant periods from the learning period and is formally known as the model’s staying power. This research investigates the effect of concept drift on the mobile gaming and iGaming industries. A benchmark paper which uses data from the mobile gaming industry is used to investigate concept drift. Additionally, the investigation of concept drift in the iGaming industry is carried out through an industrial collaboration. Three concept drift mitigation approaches are used to address the problem of concept drift being the Moving Window approach, the Incremental Window approach and the Window Dissimilarity approach. Several machine learning models are used, namely Random Forest (RF), XGBoost and Light Gradient Boosting Machine (LGBM) (amongst others). Results show that in the case of the benchmark paper, these approaches did not have any impact on performance. However, in the case of the industrial collaborator, a statistically significant improvement is noted as the RF obtains a ROC-AUC 0.741, the LGBM 0.755 and XGBoost 0.752 for the moving window approach while for the window dissimilarity approach the RF achieves 0.740, LGBM 0.754 and XGBoost 0.752. An improvement in ROC-AUC of 3% is noted when considering the results of the industrial collaborator model.