Hallucinations in Structured Extraction: A Case Study on Prompt-Based Semantic Role Labeling

Abstract

This paper investigates hallucinations in large language models from the perspective of structured extraction, using prompt-based Semantic Role Labeling (SRL) as a controlled case study. SRL requires exact span boundaries and role assignments, while prompting constrained by rationales exposes the reasoning trace associated with each prediction. Our approach employs a DSPy-based pipeline with rule-grounded signatures to show that hallucinations in constrained extraction are systematic, rationale-associated, and thus amenable to targeted mitigation through signature design. Experiments on a representative subset of the CoNLL-2012 dataset show that hallucinations persist even under constrained settings and are systematically associated with failures of rule grounding. Because each prediction is required to cite a single governing rule, errors can be organized and aggregated at the rule level without necessitating additional classification. Leveraging these observations, we introduce a Rationale-Oriented (RO) optimizer that targets such failures by selectively rewriting the relevant parts of the signature. This process mitigates systematic failure modes and leads to measurable improvements in precision, recall, and F1 across multiple evaluation settings, using a representative subset of argument roles aligned with a 5W+1H schema and no model fine-tuning. Signature-perturbation experiments show that cited rules act as diagnostic proxies rather than per-instance causal explanations: the model's parametric priors also contribute to extraction. Even so, rule-level aggregation remains a reliable predictor of where errors occur. These findings suggest that hallucination in structured tasks is closely linked to failures in instruction grounding, and that rule-level feedback and refinement offer a framework for improving reliability in prompt-based systems.