Literal to a Fault? How Large Language Models Handle Instructions That Break Codified Operational Rules

Abstract

When an operator’s instruction conflicts with a codified safety rule, an assistant should recognize the conflict and defer to the rule. This is a pragmatic problem with an objective answer: the correct action follows from a published regulation, while the difficulty is linguistic. Existing evaluations do not target this case: general safety benchmarks ask whether a model emits harmful content, instruction-following suites score obedience to formatting constraints, and work on sycophancy and rule-following shows that models can be argued out of a stated rule, but none set a user’s authoritative instruction against an external, codified rule whose deferral is objectively correct. This paper introduces a benchmark of 75 items (60 rule conflicts, 15 no-conflict controls) grounded in the International Regulations for Preventing Collisions at Sea (COLREGs) and in paraphrased International Civil Aviation Organization (ICAO) procedures, spanning four conflict framings, each presented with and without the governing rule and followed by a pushback turn. Eleven instruction-tuned models were evaluated across 1649 responses and scored by an independent language model judge (second judge Cohen’s kappa = 0.84). Because the items are author-constructed and now validated by two independent domain reviewers, compliance rates are reported as measurements on a validated benchmark rather than as mere rankings, though they remain bounded by the text scenario setting, and the model-scaling results in particular are treated as exploratory. Both model size and alignment were associated with unsafe compliance: within one family compliance fell overall from 0.51 at 0.5B to near zero at 14B, though not strictly monotonically, and at a fixed 7B size a less safety-tuned model complied far more than its aligned peers. Compliance concentrated on direct orders, fell when the rule was shown, rose under multi-turn pressure, and traded off against over-refusal, with the rule-context and multi-turn effects confirmed by item-paired tests and reported with confidence intervals. The benchmark is released for evaluating rule-aware assistants, together with the expert-validated item set and the validation materials.