Detection of outliers : a data mining approach

Abstract

Data is constantly being generated from daily life. An outlier in a set of data is an observation or a point that is considerably dissimilar or inconsistent with the remainder of the data. Outliers could potentially represent the consequential elements in the data. Analogous rules exist in where a small percentage of root causes generate a bulk of failures in networks and software. Managing to find this crucial information by sifting through the data is a very sought after exercise. Outliers can represent an error, or justifiable data which means that they can also be inspiration for further research. An outlier might enlighten researchers on an important principle or issue. Before removing outliers, researchers need to question whether that data contains valuable information that perhaps might not even relate to the intended study, but has importance in a more global sense. In this project, the Local Outlier Factor (LOF) algorithm is applied to a chosen dataset to calculate the respective computed outlier scores of each record in the chosen dataset. This involves various steps which are outlined in the documentation. The outcome of this project is to reinforce the notion that outliers shouldn't be immediately discarded and thought of as noise or errors in the data. In this project outliers will be identified for their potential to represent new and previously unexplored relationships amongst the existing attributes in the dataset.