Software engineering depends on the ability of developers to understand code, yet predicting how they do so remains an open challenge despite decades of research. Existing approaches rely either on simplified proxy measures that limit accuracy or on non-trivial measurements requiring elaborate experimental setups that are difficult to scale and apply in practice. In contrast, recent work in psychology suggests an alternative perspective: Instead of modeling task-specific phenomena directly, human behavior can be captured through cognitive regularities learned from large-scale behavioral data. This idea treats complex human behavior as the observable outcome of underlying cognitive processes that manifest consistently across tasks and domains. In this paper, we explore this perspective in the context of program comprehension. We evaluate Centaur, a foundation model trained on 160 general psychological experiments, on 9 previously published program-comprehension studies. We assess how well its predicted response distributions align with human response data and compare Centaur's performance to its base model, Llama 3.1. To better understand the source of its performance, we conduct ablation studies to isolate the contribution of different sources of information, such as the code artifacts, task-related context, and prior trials and participant responses. In a nutshell, we find that Centaur more closely aligns with human response patterns than its base model, is significantly less reliant on information from prior trials and responses, and benefits more from task-related information. These findings suggest that behavioral patterns learned from general psychological data can transfer to complex software engineering tasks such as program comprehension. More broadly, they point toward foundation models of human cognition as a basis for modeling developer behavior in software engineering.
Large language models (LLMs) are increasingly used in software-engineering tasks processing executable code and non-executable semantic cues such as comments or identifiers. These two sources can conflict when semantic cues suggest different program behavior than the code itself. It remains unclear how such semantic conflicts affect LLM behavior and which source dominates their outputs. We present the first controlled, mechanistic study of LLM behavior under semantic conflicts. To this end, we construct 45 Python snippet triplets that isolate conflicts by varying either semantic cues or implementation while keeping token-aligned pairs for causal intervention. We evaluate four open-weight LLMs on two tasks (output prediction and unit-test generation) using behavioral performance measures and residual-stream activation patching to identify token-layer states that causally contribute to behavioral differences between aligned and conflicting inputs. Our results show that semantic conflicts significantly reduce execution-grounded correctness in both tasks and that all tested LLMs often follow misleading semantic cues. Residual-stream activation patching reveals a consistent pattern for final-output prediction: The changed cue/code region and a small set of intermediate tokens carry most of the recoverable causal signal before aggregation near the output readout. For unit-test generation, this pattern extends beyond the prompt, showing that conflict-related information is recoverable at generated sites before producing expected values. Overall, our findings show that semantic conflicts affect program comprehension and downstream tasks, with relevant information concentrated in a small number of causally active residual-stream states, and demonstrate a framework for mechanistically analyzing how LLMs integrate code-related information under controlled semantic variations.
Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer et al.· 2 citations
