Discovering decisions in neural networks

Abstract

Neural networks are typically regarded as black box models due to the complexity of their hidden layers. Recent advances in classification problems using neural networks are hindered by their applicability in solving real world problems due to their lack of explainability. In critical scenarios where a simple decision is not enough, reasons to back up each decision are required and reliability comes into play. Here, we used a spatial relation dataset of geometric, language and depth features to train a neural network to predict spatial prepositions. We attempted to extract explanations by using Layerwise Relevance Propagation (LRP) on the trained model to generate relevance measures for individual inputs over positive instances. This technique redistributes relevance at each layer in the network, starting from the output and ending with the input layer relevance measures. The resulting feature relevance measures are treated as explanations as they are indicators of feature contributions towards the network’s prediction. Since explanations proved to be somewhat biased when pooling feature relevances, a baseline explanation was generated as an indicator of global input relevance for the model. Improved explanations were created by taking the difference of individual explanations from the baseline explanation to produce explanations by variation (from the baseline). The feature contribution measures obtained for each spatial preposition were qualitatively evaluated to check if explanations followed intuition. The results showed that the explanation techniques used provided different feature rankings but showed concurrence for the most relevant feature.