Classification of deceptive traits from audio-visual data

Abstract

The automated analysis and inference of human behavioural traits by machine interfaces is a growing field of research. On a smaller scope, research regarding the study of deceptive traits and classification is relatively scarce. It is known from many studies that the augmentation of multi-modal information e.g. acoustic analysis of speech, as well as visual lip-reading, can enhance the performance of speech recognition systems traditionally geared towards acoustic data only. Similarly, the analysis of speech augmented with human body language and facial patterns can help provide information on traits such as emotional state, or whether a speaker is trying to deceive an audience or interlocutor. This project investigates various applications of Machine Learning (ML) Techniques in attempt to detect deceptive actions and encapsulate those traits. Various ML Models and Hyperparameters were explored within the Hyperparameter Space with Bayesian Optimisation tuning. The model architecture used throughout this paper was LSTM-based RNN. Utilisation of the best performers played a very important role into producing the final Classifier. The proposed system is an Ensembling of two LSTM-based RNNs per the audio-visual modalities. The implemented ensembling technique utilises a Random Forest Regressor as a meta-learner between the models. This classifier achieved an AUC score of 0.607. It was trained on aligned audio-visual features. This study shows that a machine can capture and analyse deceptive traits at an adequate degree of confidence and accuracy. The existence of such a Machine Learning (ML) Classifier, suggest that there exists some patter or latent model that defines deceptive actions.