Spectrum-Based Failure Attribution for Multi-agent Systems
TL;DR
FAMAS is the first spectrum-based failure attribution approach for MASs and proposes a novel suspiciousness formula tailored to MASs, which integrates two key factor groups, namely the agent behavior group and the action behavior group, to account for the agent activation patterns and action activation patterns within the MAS execution trajectories.
Abstract
Large Language Model Powered Multi-Agent Systems (MASs) are increasingly employed to automate complex real-world tasks, such as programming and scientific discovery. While promising, MASs are not immune to defects or failures. Failure attribution in MASs, i.e., to pinpoint the specific agent actions responsible for failures, is underexplored and labor-intensive, posing significant challenges for debugging and improving MASs. To bridge this gap, we propose FAMAS, the first spectrum-based failure attribution approach for MASs. The approach performs systematic trajectory replay and abstraction, followed by spectrum analysis. Its core idea is to estimate, from variations across repeated MAS executions, the likelihood that each agent action is responsible for the failure. In particular, we propose a novel suspiciousness formula tailored to MASs, which integrates two key factor groups, namely the agent behavior group and the action behavior group, to account for the agent activation patterns and action activation patterns within the MAS execution trajectories. Extensively evaluated against 12 baselines from the Who&When benchmark, FAMAS demonstrates superior performance, outperforming all compared methods.