Minos: apparatus, parser and per-case captures for "Evaluation harness choices bias the measured tool-use capability of open-weight large language models"

Abstract

The complete artifact behind the study of harness-induced bias in tool-use evaluation of open-weight large language models: the benchmark apparatus, the model-agnostic multi-dialect call parser, the scorers, every analysis and figure script, and the raw per-case capture logs. The study follows a run-once, re-parse-many design: each (model, case) pair was run once per delivery mode and each captured raw output re-scored offline under every parser condition. Every number reported in the paper is therefore re-derivable from the released captures with no model access and no GPU. scripts/audit_paper_claims.py re-derives and checks all 141 headline claims in one command. Contents. 19 open-weight models (3B-14B) plus two held-out sets, benchmarked on three BFCL categories and an evaluation set derived from the ToolACE data, under two tool-delivery modes and three parser conditions; a 300-case generated format-stress set; a 15-task multi-turn agentic suite; and the fp16 end-to-end validation runs. Licensing. Software is MIT (see LICENSE); the capture logs, result notes, figures and task definitions are CC BY 4.0 (see LICENSE-DATA). Third-party inputs are not redistributed. The BFCL case files, the ToolACE corpus and the model tokenizers keep their own terms and are fetched from their own sources at pinned revisions by scripts/fetch_data.py, then verified against a checksum manifest. See data/README.md. Three of the tokenizers are for gated models whose licences forbid redistribution; that access barrier is itself a reported result. Redaction. The captures are the models' raw text, with one exception. In mult_native.jsonl (case multiple_61, deepseek-coder-v2-lite) the model invented a Google Custom Search call and emitted a well-formed Google API key as the key argument. Whether it synthesised that string or reproduced one memorised from training data cannot be told from the output, so the value is replaced with REDACTED_GOOGLE_API_KEY. The case already scored a miss in every condition, so no reported number depends on it and scripts/audit_paper_claims.py still reproduces all 141 claims. Start from REPRODUCE.md.