Decision Without Generation: Benchmarking a Calibrated Non-Generative Decision Model Against LLM Judges on Moderation, Hallucination, NLI and Prompt-Injection Tasks

Abstract

Large language models (LLMs) are widely used as judges for content moderation, hallucination detection and prompt-injection filtering, although they are slow, expensive and poorly calibrated when asked to verbalise probabilities. We evaluate a different design point: a commercially served decision-only model (typesafe/jev-1.13, "Jev") that returns a typed, calibrated decision for a narrow input and is billed on input tokens only. On identical items from seven public benchmarks (ToxicChat, BeaverTails, HaluEval-QA/Dialogue, BoolQ, ANLI-R3, a prompt-injection set; n=500-600 each, with Jev replicated on complete splits, 54,634 decisions), Jev has the highest AUROC of any cheap-tier LLM (GPT-5.4-nano, GPT-5-mini, Claude Haiku 4.5) on all six binary tasks, ties Haiku on ANLI, and is never significantly worse than a cheap-tier model under paired bootstrap, at 3-63x lower cost and 3.7-8.6x lower latency. Against frontier models run as equal-cost classifiers, Jev is significantly better than GPT-5.4 (reasoning off) on 4 tasks and tied on 3, one of which (HaluEval-QA, -0.013 AUROC) is a borderline loss; Claude Sonnet 4.6 (42-97x the cost) beats Jev on ANLI-R3 and dialogue-grounded hallucination. Jev's raw probabilities have lower Brier than every LLM's; after cross-fitted Platt recalibration the gap persists against the cheap tier and GPT-5.4 but closes against Sonnet. A quantisation control shows the ranking advantage is not an artefact of the LLMs' coarser verbalised probabilities. We report a negative result - no cheap-tier cascade on Jev's uncertain band significantly improves on Jev alone - and two self-caught evaluation artefacts. Principal limitations: public benchmarks that the model may have seen in training; a single-vendor black box; no same-item specialist classifier baselines. Technical report (preprint), version 1.0.0. Generative AI use: Fable 5.1 was used for manuscript drafting and experimental code generation. OpenAI Codex assisted with preparing the public release, documenting the repository and checking offline reproducibility of the archived numerical results. The author takes responsibility for the report and its results. Author affiliation: Canly Inc.; Chief AI Officer (CAIO). Correspondence: hagino@apache.org. Institutional contact: t.hagino@can-ly.com (CAIO, 株式会社カンリー / Canly Inc.). See the manuscript Disclosure section for the confirmed conflict-of-interest and self-funding declaration. Software and reproducibility archive: 10.5281/zenodo.22885409. Development repository: haginot/decision-without-generation.