PREreview of "COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/23177551. ## Summary Routing picks one model and stops; dense collaboration invokes peers on every query. COMED argues both ends of this spectrum are wrong and formalizes the middle: collaboration is non-monotonic — peers can rescue failures no single model solves alone, but they can also corrupt initially correct answers. The paper introduces a post-anchor controller: an anchor model answers first, then the system chooses among Accept (return the answer), Verify (lightweight peer checking), or Collaborate (full cross-model deliberation). The decision uses three signals — anchor self-consistency over N reasoning paths, the router's top-1/top-2 margin, and a confidence-gated peer probe that checks for credible disagreement. The trade-off is formalized as a rescue–harm decomposition: selective collaboration wins when rescued errors outweigh collaboration-induced harms. Across two open-weight 7–8B pools (Medical-4, Bal-3), four benchmark families, and fixed/router anchor regimes, COMED improves its anchor in all 16 settings (average +4.57pp, up to +10.7pp on MedQA), invoking at most two models per query with ~33% fewer decoded tokens than dense Always Collab. On HLE with frontier models, it lifts GPT-5.5 from 23.13% to 28.13%, outperforming dense collaboration. ## Strengths 1. **It names and formalizes a real, under-discussed phenomenon.** Everyone running multi-model systems has watched a peer "correction" make a good answer worse; this paper turns that folk observation into a measured decomposition. The rescue–harm framing is the kind of lens that outlives the specific controller — Always Collab being net-negative in 10 of 16 settings is a result the field needed stated plainly. 2. **The experimental design isolates the actual claim.** Matched controls separate query selection from peer selection: at identical collaboration rates, COMED beats Always-Collab Top-1 by up to 9.1pp and Random same-rate Top-1 by up to 6.4pp, so the gains come from *when* to collaborate, not from picking a better peer. The Verify-gate ablation is load-bearing — removing it raises harmful flip-downs from 4.1 to 7.7pp and token cost from 5.4k to 8.6k on MedQA — which is exactly the evidence a practitioner needs that the gate earns its keep. 3. **Honest about the failure mode it can't fix.** The limitations section states plainly that COMED mitigates but does not eliminate collaboration harm, that dollar-denominated cost is unevaluated, and that latency measurements are preliminary. A paper whose core claim is "collaboration is risky" disclosing its own residual risk is intellectually consistent. ## Major concerns 1. **The "cheap" triage gate is the anchor multiplied.** Self-consistency over N reasoning paths is the primary gate signal, which means every query pays N anchor generations before any Accept/Verify/Collaborate decision. The paper's cost accounting is in decoded tokens for the collaboration path, but the SC sampling is fixed overhead on 100% of queries — the triage itself is the most expensive component, and it never appears in the cost framing. 2. **No mixed-scale pools — the setting where this matters most.** All open-weight results use comparable 7–8B models. Production routing is small→frontier, where the rescue opportunity (and the corruption risk, and the price of being wrong) is largest. The HLE stress test is 160 examples with 70% escalation at frontier prices and no dollar accounting — the one regime where the rescue–harm trade-off has real money attached is the least measured. 3. **The peer probe is a model call whose own errors go uncharacterized.** Verify hinges on a "targeted peer probe" detecting credible disagreement, but the probe's false-positive/false-negative rates aren't reported — a probe that cries wolf drives unnecessary collaboration, a probe that misses conflict lets errors through. And probe cost counts toward token cost but not the "escalation rate," a definitional choice that flatters the headline numbers. ## Practitioner perspective The durable contribution is the rescue–harm accounting, not the controller: before adding any peer review to a serving path, measure your own rescue vs. harm rates on production traffic, because the 10-of-16 net-negative finding for dense collaboration should scare anyone running debate-by-default. The pattern I'd steal is "verify before collaborate" — a cheap targeted check as a gate — but I'd want it gated on something cheaper than N-path self-consistency, and I'd want the whole thing priced in dollars per query on a mixed-scale pool before it touches a production path. ## Overall assessment A genuinely insightful paper that formalizes the missing middle between routing and collaboration, with rigorous matched controls and admirable honesty about residual risk. Recommend with revisions: account for the self-consistency overhead in the cost story, test on mixed-scale pools with dollar-denominated cost, and characterize the peer probe's own error rates. Competing interests The author declares that they have no competing interests. Use of Artificial Intelligence (AI) The author declares that they used generative AI to come up with new ideas for their review.