Determining the Relevance of Features for Deep Neural Networks

Christian Reimers, Jakob Runge, Joachim Denzler ;

Deep neural networks are tremendously successful in many applications, but end-to-end trained networks often result in hard to understand black-box classifiers or predictors. In this work, we present a novel method to identify whether a specific feature is relevant to a classifier’s decision or not. This relevance is determined at the level of the learned mapping, instead of for a single example. The approach does neither need retraining of the network nor information on intermediate results or gradients. The key idea of our approach builds upon concepts from causal inference. We interpret machine learning in a structural causal model and use Reichenbach’s common cause principle to infer whether a feature is relevant. We demonstrate empirically that the method is able to successfully evaluate the relevance of given features on three real-life data sets, namely MS COCO, CUB200 and HAM10000."