Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

Abstract

Graph neural networks dominate recent work on microservice root-cause analysis, yet recent results question whether the graph contributes. Those results compare whole pipelines, so when a flat model wins one cannot tell whether structure is useless or redundant. We run the comparison they imply on RCAEval: three learned arms with identical features, optimiser, validation split, early-stopping rule and scoring head, in which a single neighbour-mixing term separates the graph arms. Across two RCAEval benchmarks, two topology sources and four regimes we find no reliable graph-specific effect: in-distribution the graph model leads the conventional flat model by 0.003 Avg@5 (p = 0.844, n = 6 disjoint folds). Auditing the pipeline surfaced two benchmark properties that condition any result on it. RCAEval injects faults into only five services per system while exposing 12-70 in telemetry, and the headline metric is Avg@5: a ranker reading no telemetry at all places the true culprit in the top five on 99.7% of held-out incidents, scoring Avg@5 0.488. That prior, not the uniform-random 0.137, is the honest in-distribution floor, and it collapses to 0.192 across systems. The second property is a non-uniform column schema that silently zeroes telemetry for most RE1 cases, which changes how any reimplementation of the benchmark should be read. We reproduce a published baseline (BARO), RCAEval's own reference implementation; on the one system with a clean schema it reaches similar aggregate accuracy to our own "naive" heuristic, within 0.004, under a different scoring rule, and we read the divergence elsewhere as a schema effect rather than a quality difference. The audit motivated a new model. PSC-GRCA separates a candidate score into a system prior, telemetry evidence, and a centred graph residual, and scores each term separately. On the six fixed stratified folds it reaches mean Avg@5 0.915 against 0.864 for a flat MLP and 0.862 for a capacity-matched no-neighbour control, a fold-level paired Wilcoxon p = 0.03125, the exact two-sided minimum at n = 6. Under transfer it reaches 0.747 against 0.671 for the MLP. Ablations locate the in-distribution gain in the prior term: prior-only scores 0.488, prior-free drops to 0.850, and the graph residual alone reaches 0.800, so structure adds little on its own. The prior-swap penalty is a guard rather than the mechanism: the swap passes are made deterministic, and because the evidence and residual branches read no prior features the penalty is then exactly zero, so the no-swap runs reproduce the swapped ones. These results are exploratory: three systems, one architecture family, and a single prior feature. We close with a twelve-item checklist for this class of study, distilled from sixty-two defects recorded while producing the results above, each item motivated by a failure it would have caught.