Harnessing Driver Metacognition to Advance In-Context Learning for Autonomous Vehicle Social Intelligence

Abstract

This conceptual paper explores the integration of driver-based social metacognition experiments with prompt engineering for in-context learning, aiming to enhance the social intelligence of autonomous vehicles (AVs) in pedestrian interactions. We propose a novel framework that leverages human metacognitive processes to address limitations in the social cognitive capabilities of large language models (LLMs) when applied to AV decision-making systems.While AVs have made significant technological strides, their ability to navigate interactions with vulnerable road users remains a critical challenge in urban environments. These scenarios demand advanced reasoning capabilities that traditional AI systems often lack. Foundation models like LLMs offer a promising solution for equipping AVs with enhanced common sense and social reasoning skills. However, despite their remarkable language understanding and generation abilities, LLMs are trained on complex language patterns and lack the nuanced social intelligence derived from real-world interactions and biological foundations.To overcome this gap, we propose using driver-based metacognition experiments to generate effective prompts for in-context learning. These prompts aim to capture the core of human social cognition, focusing on two critical aspects: improving the accuracy of social cognitive estimations and calibrating confidence levels in these estimations. By distilling drivers' implicit and explicit metacognitive processes into structured prompts, our approach guides LLMs in modeling pedestrian intentions and beliefs across diverse traffic scenarios.This framework represents a step forward in advancing AV systems, enabling safer and more intelligent navigation of complex social environments by synthesizing human cognition with AI reasoning.