Summary Benchmarks for policy-adherent language agents typically split tasks into those where the agent must refuse and those where it must act, and report accuracy on each. Such designs measure the joint effect of task difficulty and of whatever conversational dynamics the benchmark contains, without separating them. DF-Core is a stipulated-rule benchmark in which the same 56 decision families are evaluated both without conversational pressure and under a fixed three-turn pressure sequence, in each of two directions: pressure to act despite a blocking condition, and pressure to withhold a required action. Because the oracle is a deterministic function of an explicitly stated rule and an explicitly stated variable state, correctness is decidable without human annotation or an LLM judge. Main finding Across 4 models and 3 independent replication runs (4,704 scored responses), we find a small baseline directional asymmetry (14.6 percentage points on the unperturbed condition) and a much larger pressure-induced one. Measured within the same multi-turn conversations, on the same tasks: accuracy falls 57.2 pp under pressure toward inaction, versus 6.7 pp under pressure toward action (92.96% vs 28.47% pooled across pressure turns, two-proportion z = 29.6). Errors run 4.8:1 toward false blocking — the model refuses an action the stated rule requires. The effect appears in all four models and all three runs, but its magnitude varies substantially by model (gap 42.5 pp to 92.9 pp). Scope of the claim This is a bounded replication-and-extension of an actively studied phenomenon, not the discovery of a new one. Directional compliance asymmetry under user pushback has been measured by several groups in 2026; the paper positions this work explicitly against that literature (τ-bench / τ²-bench family, PolicyGuard, compliance-asymmetry metrics, over-refusal benchmarks). The specific contributions claimed are: (i) a design that separates baseline task-difficulty asymmetry from pressure-induced asymmetry by holding the task fixed and toggling pressure within the same conversation; (ii) a deterministic oracle removing human and LLM judgment from scoring; (iii) replication across three independent runs with the full artefact released. Results are not prevalence estimates for deployed systems. Cases are synthetic and stipulated. No frontier-class model was tested, and the most capable model in the set is also the most robust — the asymmetry may shrink with scale. See §7 of the paper for the full limitations discussion. Package contents df_core_benchmark_56_v1_3.csv — the benchmark: 56 case families across 14 domains, balanced 28/28 on direction, with governing rules, dependency variables, baseline states, pressure-turn texts and oracle labels build_prompts.py — expands the benchmark into a flat conversation file (deterministic, no API calls) df_core_pilot_copy_prompts.csv — the exact prompts sent run_benchmark.py — API runner with resume support and pre-flight cost estimate df_core_api_results_run1/2/3.csv — all raw results, 4,704 scored responses including full unedited model outputs analyze_results.py — regenerates every table in the paper from the raw data; output is tagged with the corresponding paper table Every number in the paper can be verified without API access or cost by running python analyze_results.py on the included data. See README.md in the package for step-by-step instructions.
Michał Nowak· Zenodo (CERN European Organi...· 0 citations
Summary Benchmarks for policy-adherent language agents typically split tasks into those where the agent must refuse and those where it must act, and report accuracy on each. Such designs measure the joint effect of task difficulty and of whatever conversational dynamics the benchmark contains, without separating them. DF-Core is a stipulated-rule benchmark in which the same 56 decision families are evaluated both without conversational pressure and under a fixed three-turn pressure sequence, in each of two directions: pressure to act despite a blocking condition, and pressure to withhold a required action. Because the oracle is a deterministic function of an explicitly stated rule and an explicitly stated variable state, correctness is decidable without human annotation or an LLM judge. Main finding Across 4 models and 3 independent replication runs (4,704 scored responses), we find a small baseline directional asymmetry (14.6 percentage points on the unperturbed condition) and a much larger pressure-induced one. Measured within the same multi-turn conversations, on the same tasks: accuracy falls 57.2 pp under pressure toward inaction, versus 6.7 pp under pressure toward action (92.96% vs 28.47% pooled across pressure turns, two-proportion z = 29.6). Errors run 4.8:1 toward false blocking — the model refuses an action the stated rule requires. The effect appears in all four models and all three runs, but its magnitude varies substantially by model (gap 42.5 pp to 92.9 pp). Scope of the claim This is a bounded replication-and-extension of an actively studied phenomenon, not the discovery of a new one. Directional compliance asymmetry under user pushback has been measured by several groups in 2026; the paper positions this work explicitly against that literature (τ-bench / τ²-bench family, PolicyGuard, compliance-asymmetry metrics, over-refusal benchmarks). The specific contributions claimed are: (i) a design that separates baseline task-difficulty asymmetry from pressure-induced asymmetry by holding the task fixed and toggling pressure within the same conversation; (ii) a deterministic oracle removing human and LLM judgment from scoring; (iii) replication across three independent runs with the full artefact released. Results are not prevalence estimates for deployed systems. Cases are synthetic and stipulated. No frontier-class model was tested, and the most capable model in the set is also the most robust — the asymmetry may shrink with scale. See §7 of the paper for the full limitations discussion. Package contents df_core_benchmark_56_v1_3.csv — the benchmark: 56 case families across 14 domains, balanced 28/28 on direction, with governing rules, dependency variables, baseline states, pressure-turn texts and oracle labels build_prompts.py — expands the benchmark into a flat conversation file (deterministic, no API calls) df_core_pilot_copy_prompts.csv — the exact prompts sent run_benchmark.py — API runner with resume support and pre-flight cost estimate df_core_api_results_run1/2/3.csv — all raw results, 4,704 scored responses including full unedited model outputs analyze_results.py — regenerates every table in the paper from the raw data; output is tagged with the corresponding paper table Every number in the paper can be verified without API access or cost by running python analyze_results.py on the included data. See README.md in the package for step-by-step instructions.
Michał Nowak· Zenodo (CERN European Organi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
