Imputation and anomaly detection for consumer load profiles

Abstract

The advent of smart meters has opened new possibilities for utilities. Billing is no longer the sole function of the meter. The load profile data, registered by the smart meters, can be analysed in order to obtain knowledge about various aspects. It can be used to indicate stress points in the network, calculate technical losses and explore methods for their reduction. In reality, though, it is inevitable that a certain amount of data will get lost. While doing every effort to keep this loss of data to the minimum possible, the missing portions of data must be imputed before any further analytical activity can be performed. This thesis explores techniques for imputing such missing data in smart meter load profiles. The proposed method implements a k-nearest neighbors approach where the imputed part is calculated by searching the past consumption of the consumer for patterns that best resemble the portion around the missing part and taking an average of the parts which corresponds to the missing portion. The length of each segment and the number of segments to be considered are arbitrary and therefore suitable values had to be determined through a tuning process. Using the developed algorithm on a sample of 335 consumer load profiles, the average RMSE was 7.47% of the actual values. An area of great concern for utilities is non-technical losses which can be made up of billing inaccuracies, faulty meters and fraudulent consumers. This thesis develops a method of anomaly detection for finding consumers with irregular behaviour which are likely to contribute to non-technical losses. The consumers are grouped into clusters having similar weekly consumption behaviour by using hierarchical clustering. For each consumer two novel coefficients are computed: the Anomaly Coefficient, which is a measure of how different the consumption behaviour of the consumer is from the other consumers in the same cluster, and the Cluster Change Coefficient, which is a measure of how irregular in consumption behaviour the consumer is, compared to all the other consumers. Consumers having high values for any, or both, of these coefficients are more likely to exhibit non-technical losses.