TSAPLAY : A framework to aid reproducibility in and exploration of the field of targeted sentiment analysis

Abstract

The proliferation of platforms online which enable users from around to world to voice their opinions on any subject over time has led to the emergence of the largest publically accessible textual representation of public opinion which has not gone unnoticed by the NLP community, using this data in conjunction with sophisticated models for various tasks. Targeted Sentiment Analysis (TSA), whereby the sentiment polarity towards a particular target is identified, presents itself as one of the most popular of such tasks garnering a wide range of different approaches over the years. In this work, we attempt to recreate some of the most seminal approaches to this task in an effort to evaluate the current state of reproducibility of this field, and whether it has been neglected by the fields’ rapid growth. Towards this end, we develop a framework which facilitates TSA model research by decomposing the various parts of the Machine Learning (ML) pipeline into separate modules, abstracting lower-level complexities from the end-user while exposing entry points for feature-extendibility and, providing an ideal environment where models can be evaluated and compared more efficiently. Using this framework to investigate reproducibility, we identify three issues, namely, lack of specificity, the importance of multiple experimentation runs and, the use of misleading metrics which do not account for class distribution in datasets. Finally, we evaluate different out-of-vocabulary clustering approaches and find downstream effects which merit further investigation.