Calibrated Decisions Are Not Calibrated Probabilities: An Exact-Target Audit of Jev and Three Open Decision Models

Abstract

Decision models answer typed questions with probabilities instead of text, and their main selling point is that those probabilities can be trusted. TypeSafe says its Jev model, trained with an unpublished method called Reinforcement Learning for Calibrated Decisions (RLCD), returns "epistemically honest" probabilities. I audit that claim on cases where the right probability is known exactly, and on data where many people voted on each item. Several concurrent studies, posted in the two weeks after my first measurements, also find that Jev's selection primitive (Choice) is overconfident while its yes/no primitive (Noul) is not. My results agree with theirs and add a controlled breakdown. With no evidence, Choice puts 0.83–0.93 on heads for a fair coin. The pull comes from the word "heads" itself, not from the option key or list position, and one 55%-reliable witness shrinks it from +0.35 to +0.06. When the evidence states a probability, Choice behaves like a switch: moving a stated base rate from 45% to 55% raises it by 0.59–0.89 depending on wording, where the correct change is 0.10. Noul stays within 0.014–0.029 of the stated rate. Asking one Noul per option and normalizing cuts held-out Brier error from 0.067 to 0.003. After temperature scaling, Choice is still 55× worse on stated base rates, and even a flexible monotone map leaves it 21× worse. On 650 real items with 100 or more human votes each (ChaosNLI and DICES-350), Noul is closer to the human distribution than Choice. But neither beats a uniform guess on the most contested items, and one fitted temperature makes the two equivalent. Three open checkpoints that call themselves RLCD models fail the audit in different ways, and a rebuild of one model's pre-RL stage shows that its cue saturation predates RL. Jev's Choice probability is better read as a decision score than as a probability. The core audit costs under $0.10 in API calls and reproduced unchanged 12 days later. Code, prompts, raw responses and pre-registrations: https://github.com/MohitSV/jev-calibration-audit