Peering inside the 'Black Box': understanding and refining deep neural networks with representational similarity analysis

Abstract

Deep neural networks, and Transformer models in particular, have achieved unprecedented success in natural language processing tasks. Despite this success, they are infamous for their status as black boxes. Specific details on how they encode and process high-level linguistic task-relevant information remain difficult to characterise by humans. This hinders trust and reliability, particularly in critical applications like clinical text processing. This thesis seeks to tackle the interpretability problem by applying analytic methods inspired by cognitive science, with the overall goal to build a deeper understanding of how these models represent and process language, and suggest model interventions based on these findings. Two primary research questions guide this investigation: how do Transformer models represent and process linguistic features, and how do variations in model architecture, training data, or fine-tuning influence these representations? The thesis comprises varied technical chapters with a motivational thread of linguistic interpretability. The work begins with an exploration into interpretability within the clinical domain, developing a perturbative technique to identify diagnostically influential sentences in clinical letters. I then transition to more advanced interpretability techniques, introducing Representational Similarity Analysis and linear probing. A particular focus is placed on layer-wise analysis to observe how salient linguistic signals are encoded throughout the network. Expanding on this, I use these techniques to measure the fine-grained representation of noun-noun compound thematic relations. This toolkit is then applied to Irish morphosyntax, comparing monolingual and multilingual models to demonstrate the monolingual model's stronger encoding of specific linguistic phenomena. Finally, I introduce Representational Similarity Regularisation, a novel approach to inducing alignment between representations and a target signal. This method aligns models with cognitive signals elicited by natural language, improving performance on semantic textual similarity tasks. This thesis provides novel insights into Transformer models and explores using cognitive signals to build more robust human-aligned neural networks.