Reading User Distress From the Inside: Do a Language Model's Hidden States Track a Person's Distress Better Than the Model's Own Reply Does?
When a person tells an AI assistant about something painful, two things happeninside the model that we can measure separately. First, the model builds aninternal reading of how distressed the person is; this reading can be recovered fromits hidden states with a simple trained readout. Second, the model produces a reply...