Skip to content

Evaluating and Benchmarking the System One Model Jev

Sep 2026 · 33 citations · ⚡ 7 influential · 77 references
Computer Science

TL;DR

Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder.

Abstract

Jev is a commercial"System One"model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true, with probabilities described as calibrated. Such models target small decisions in information access pipelines. We evaluate Jev (jev-1.13.0) zero-shot on 37 datasets, from classification and routing to moderation and rubric scoring, with one frozen template per dataset and full evaluation splits: 346,009 requests for under US$10. For reference, we score Qwen3.8-27B and Gemma-4-E4B on identical requests via their exact option probabilities. Jev reaches 95-99% accuracy on IMDB, SST-2, HellaSwag and ARC, beats Qwen on 27 of 37 datasets, with none of Qwen's leads outside the bootstrap intervals, and Gemma on all 37, and its choice probabilities are well calibrated and support selective prediction. All three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based judgments, and binary probabilities need tuned thresholds, which raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions, unlike both open models; probes rule out shallow memorization but not memorized question-answer pairs. Extending the protocol to sequential decisions, where the model chooses the next action at every step, Jev completes 35% of ALFWorld's unseen tasks, 42% of 90 TextWorld games and 21% of BabyAI-Text missions, chooses the optimal Blackjack action in 90% of decisions, and on FrozenLake almost never moves away from the goal to avoid holes. It often repeats actions that had no effect. Qwen completes more ALFWorld and TextWorld games than Jev (54.5% and 51.1%) but shares its failures on BabyAI-Text and FrozenLake. We release the code, harness and all raw responses.

View source

Similar papers

#natural language process... Preprint Oct 2026

HakemBench: A Turkish Benchmark of Typed Decisions

HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-chec...

Sait Furkan Teke · 1 citation
#artificial intelligence Preprint Oct 2026

Labels Override Definitions in Jev-Style Typed Decision Models

A two-call test is given that tells a practitioner which case applies to their model, and what four mitigations are worth, and a preference for the label the authors call option-label bias is measured.

Seyedarmin Azizi, Erfan Baghaei Potraghloo, M. Pedram · 5 citations
#artificial intelligence Preprint Sep 2026

CruxBench: A Benchmark of Information Discovery

CruxBench is introduced, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question and finds that VOI correlates highly with independent measures of model capability and captures cruxes'usefulness for answering...

Hui Dai, Li-Na Piao, Nick Merrill et al. · 0 citations
Preprint Sep 2026

JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges

Security evaluation of tool-using agents requires judging actions in context, yet generative judges add latency, explanation overhead, and output-validation failures. We study whether JEV, a typed decision model, offers a useful alternative for retrospective trace classification. We evaluate JEV and four generative jud...

Zhi-Qiang Wang, Yi-Chao Gao · 5 citations · ⚡2
#artificial intelligence Preprint Aug 2026

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

This work introduces PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models, and measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model believes.

A. Chetvergov, Mikhail Solovev, Timofei Sivoraksha et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.