Agentic Explainable Artificial Intelligence (Agentic XAI) Approach To Explore Better Explanation: A Case Study in Decision Support for Rice Cultivation in Japan

Abstract

Explainable artificial intelligence (XAI) reveals how explanatory variables relate to a response variable, yet communicating XAI outputs to laypersons remains difficult, limiting trust in AI-based predictions. Large language models (LLMs) can translate technical explanations into accessible narratives, but iterative refinement of XAI explanations by an autonomous LLM agent remains unexplored. This study proposes an agentic XAI framework that combines SHapley Additive exPlanations (SHAP) with iterative refinement by a multimodal LLM and tests it as an agricultural recommendation system on rice yield data from 28 fields in Japan. From a SHAP result, the agent explored additional analyses across 11 refinement rounds (Rounds 0-10). Crop scientists (n = 12) and LLM judges (n = 14) scored every round on seven criteria: Specificity, Clarity, Conciseness, Practicality, Contextual Relevance, Cost Consideration, and Crop Science Credibility. Both groups found that refinement raised the average score by 30-33% over Round 0, peaking at Rounds 3-4, after which quality declined, below the starting point for crop scientists. Refinement therefore requires strategic early stopping, which challenges assumptions of monotonic improvement. Criterion-level trajectories indicate a bias-variance trade-off. Early rounds lacked Specificity (bias), whereas excessive iteration eroded Conciseness and raised Cost Consideration through ungrounded economic reasoning (variance). The LLM judges overscored every criterion by 1.4-2.3 points but largely preserved the experts' ranking of rounds (Spearman $\rho$ = 0.58-0.90), so screened LLM judges can flag the quality peak despite unreliable absolute scores. Trustworthy agentic XAI also needs expert-anchored screening of LLM judges and transparent, verifiable refinement records.