AiDER: Auditing and Document Evaluation via Rule Compilation and Small Language Models – An Education Case Study

Abstract

Reviewing documents against natural-language requirements is hard when those requirements demand interpretation, contextual evidence, and non-trivial judgment. It is even harder when the review is to be automated reliably and made auditable. In domains like education, where teaching materials must align with pedagogical models and evolving expectations, including the rise of Large Language Models (LLMs), such decisions cannot be reduced to mechanical checks. When an LLM assigns a verdict directly, interpretation, evidence selection, and judgment collapse into a single opaque step, making errors hard to localize and outcomes hard to audit. AiDER separates interpretation from judgment under a deliberate 2B-4B parameter constraint. Each natural-language requirement is compiled into an executable Common Expression Language (CEL) rule, and static analysis of that rule produces a bounded-evidence scaffold that constrains what the model may extract from the document. The verdict is then computed deterministically by applying the rule to the extracted evidence (no LLM call at this step), yielding an inspectable trace from requirement to rule, rule to evidence, and evidence to verdict. We evaluate AiDER on four small models (Gemma 4 and Qwen3.5, at 2B and 4B) under three compilation strategies: human-written reference rules, direct compilation, and validation-loop repair. The evaluation uses 200 requirement-document pairs from a higher-education course pack. Across three performance dimensions (syntactic validity, semantic fidelity, and document-level agreement), direct compilation often yields valid but semantically divergent rules, whereas validation-driven repair substantially improves both validity and fidelity across all four models. An ablation shows that the static scaffold alone encodes most of the extraction intent: LLM-generated extraction contracts add little when the rule is well-specified, but matter more when it is ambiguous.