Exploring word embeddings using explainable artificial intelligence

Abstract

Sentiment classification tasks are usually based on extracting textual features from the provided data. It can either be achieved through manual processes such as annotating words in a sentence as content words or high-frequency words and use pattern-matching to detect the correct pragmatic import portrayed by the author. More advanced machine-learning techniques use deep-learning to extract features from text automatically. During the research, we observed how the majority of deep-learning sentiment analysis research is based on obtaining a high level of accuracy with a custom-build machine-learning model. The researchers would not delve deeply into why they chose a particular type of word embedding over another. This study aims to benchmark word embeddings against each other using a Twitter dataset. Another problem that inspired this study is the lack of transparency with a deep-learning classifier’s classifications. In this work, we also seek to explore Explainable Artificial Intelligence (AI) (Explainable AI (xAI)) in the specific scenario of sentiment classification. We use the research performed by Son et al. as a benchmark for classifier performance, however, taking a different approach. We compare and contrast various word embedding sets to rank them based on their performance on a Twitter-based dataset and use xAI techniques to evaluate the model when facing unseen data. xAI is a new analytical concept designed to expose the features learnt by a machine-learning model on a particular training dataset. The xAI technique is based on the evaluation carried out by Samek et al., which analyses the robustness of a model by removing the most prominent features from the corpus and observing the decline in accuracy, achieved with interpretability analysis. In this study, this is performed for all the word embedding types tested, thus exposing the most resilient word embeddings by observing which type performs best despite the loss of features. A benchmark was carried out between five-word embedding types: Embeddings from Language Models (ELMo), Global Vectors for Word Representation (GloVe), fastText, Word2Vec (W2V) Continuous Bag-Of-Words (CBOW) and W2V SkipGram (SG). The accuracy and F1-score were compared, and the top-performing word embeddings were the GloVe Crawl 840B variant, achieving the global best F1-score of 97.72%. A robustness test was carried out by removing prominent features from the dataset 50 times and observing the decline in accuracy for each word embedding type. FastText word embeddings suffered the least accuracy drop when the prominent features were removed. Apart from these results, an analysis of the prominent training features was carried out, aggregating each feature’s sentiment scores and listing the top five from positive and negative sentiment poles. The analysis exposed the natural bias in the dataset towards Twitter hashtags, listing it as the top negative feature.