Predicting facial impressions using guided Grad-CAM and a large language model

Abstract

Facial impressions are central to social interaction, shaping judgments of attractiveness, dominance, and trustworthiness. Although psychophysical research has identified factors that contribute to these impressions, computational methods, particularly deep learning, offer a complementary, data-driven identification of impression-relevant facial cues. We trained a Visual Geometry Group 19-layer (VGG19) regression model to predict impression ratings from facial images and used explainable AI methods to interpret its predictions. We generated Grad-CAM and Guided Grad-CAM visualizations to identify decision-relevant facial regions at coarse and fine-grained levels. We used a large language model to produce structured, reproducible linguistic summaries of Guided Grad-CAM outputs and quantitatively cross-checked these summaries using an independent coarse region-of-interest (ROI) analysis that partitioned heatmap activation across the eyes, nose, mouth, and the remaining facial area. The results indicate impression-specific activation patterns. Attractiveness showed distributed emphasis across central facial regions, dominance involved balanced contributions with stable upper-face cues, and trustworthiness was characterized by strong mouth- and eye-related activation. The language-based summaries were repeatedly consistent with ROI-level distributions, supporting their use as constrained descriptions of salient activation patterns. This study demonstrated a practical workflow that integrated visualization, structured language outputs, and quantitative cross-checking to improve interpretability in facial impression prediction models.