Hybrid machine learning–statistical method for anomaly detection in flight data

Abstract

This paper investigates the use of an unsupervised hybrid statistical–local outlier factor algorithm to detect anomalies in time-series flight data. Flight data analysis is an activity carried out by airlines primarily as a means of improving the safety and operation of their fleet. Traditionally, this is performed by checking exceedances in pre-set limits to the flight data parameters. However, this method highlights single events during a flight, making this analysis laborious. The process also fails to establish trends or reflect potential unknown hazards. This research took advantage of machine learning techniques to recognize patterns in large datasets by implementing the local outlier factor (LOF). In order to minimize human input, a statistical approach was adopted to establish the threshold value above which the flights are considered to be anomalous and interpret the scores. This paper shows that LOF quantifies the degree of outlier-ness of an outlier rather than binary categorizing a point into inlier or outlier, as in the case of clustering algorithms. Thus, with LOF, for the first time, we demonstrated that in the aviation industry, anomalous flights could not only be identified but also be given an anomaly score to compare two anomalous flights in an unsupervised manner. Furthermore, LOF helps to track anomalous behavior in time during the flight. This is insightful when a flight is abnormal, only for some seconds or short duration. For the first time, we attempted to detect flight parameters responsible for anomalous behavior or at least give direction to human experts looking for the cause of abnormal behavior. This was all analyzed with real-life flight data in an unsupervised manner in contrast to simulated data.