Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes
Abstract
Recent work shows that Large Language Models (LLMs) encode more information in their activations than they show in their responses. We ask whether the information contained in activations could be used as a source of measurement for the presence of concepts. We compare two algorithms for extraction of concepts, i.e., Recursive Feature Machine (RFM) algorithm and linear probing against an embedding baseline, surface baselines, and the model's own response using a human-annotated Environmental, Social and Governance (ESG) dataset of financial texts. The best linear probe ranges from 0.6 to 3.3 percentage points below fine-tuned domain classifiers. The probes also outscore the model's own response in all comparisons. Furthermore, the probes outperform the RFM concept vectors in all accuracy comparisons. In a synthetic mixture test, their mean-pooled score tracks the number of concept sentences in a paragraph better than the concept-vector score. Put together, a linear probe on a frozen LLM is a cost-effective and generalizable alternative to fine-tuning a domain classifier.
