Executable explanation traces for legal LLM predictions via retrieval-augmented codification
Explainability is particularly challenging in legal prediction, where decisions are expected not only to be accurate but also to be justified under explicit legal norms and open to external scrutiny. Legal reasoning therefore provides one of the most demanding settings for explainable AI because a legal determination often depends on information distributed across the input record, external legal sources, procedural context, evidentiary assumptions, and institutional judgment. Direct outcome generation hides this decision path, while natural-language rationales and retrieved citations do not necessarily show whether a legal condition was actually applied. This article presents studies of retrieval-augmented codification as a way to produce executable explanation traces for legal large language model (LLM) predictions. The goal is not to generate human-facing rationales directly, but to produce auditable artifacts: short programs that extract signals, apply program-level conditions, follow branches, and produce final labels under a fixed output schema. The approach retrieves legal sources and fold-safe in-domain examples, compiles them into an executable intermediate representation, and refines the program using model feedback. The resulting trace records retrieved materials, generated conditions, feedback, and the final prediction. We evaluate this framework on five legal benchmarks covering Chinese criminal law, European human-rights cases, U.S. class-action complaints, Indian judgments, and Japanese tort cases. The evaluation asks not only whether the predicted label is correct, but also whether the explanation is executable, whether its conditions are supported by retrieved legal sources, whether it responds to legally material edits, and where it fails. The results show that executable explanations can be obtained without sacrificing predictive utility: the full codification setting improves mean label accuracy over direct prompting, chain-of-thought prompting, one-pass codification, and text-based retrieval baselines. More importantly, trace analysis reveals why legal explainability is difficult. Codification is most informative when decisions depend on definitions, thresholds, exceptions, and other rule-like components. It is less complete when benchmark labels depend on procedural, evidentiary, or discretionary factors outside the input record. These findings position retrieval-augmented codification as a practical explainable artificial intelligence (XAI) tool for studying when legal LLM predictions can be explained through explicit, source-supported decision paths.