Rubrica v0.1.2: software description and validation report

Abstract

This report documents Rubrica v0.1.2, a local application for grading scanned, handwritten exams against an instructor's rubric. Under the default local-Ollama configuration, cover-page extraction runs on the instructor's machine. Stored names, SIDs, and page 0 are excluded from external grading requests; those requests contain an anonymous identifier, answer-page images, and rubric text. Rubrica does not inspect answer pages or rubrics for incidental identifiers, so instructors should review and remove that information before external grading. Version 0.1.2 is the pilot build, a signed and notarized standalone macOS application (Tauri shell around the Flask backend, React interface), distributed to the first outside testers in late August 2026. This report contains a software description, a validation note, architecture and grading-pipeline diagrams, a README, and citation metadata. It does not contain source code or an executable. The deposited documents and diagrams are released under CC BY 4.0. That license does not cover the absent AGPL core source or proprietary pilot application. Validation evidence. An author-run, matched-input cross-model comparison (April 2026; 30 exams, 840 scored items; Claude Sonnet 4.6 vs. o4-mini) reported ICC(3,1) 0.964, quadratic weighted kappa 0.883 on letter grades, 96.2% within-one-point agreement, and mean absolute error of 0.15 points per question. The observed production-minus-reference mean difference was +0.64 points per exam; its approximate 95% confidence interval spans zero. The comparison estimates cross-model concordance and serves as a structural check, not a third-party audit or human-accuracy study. A July 2026 comparison against human-assigned reference scores covered one 25-exam macroeconomics midterm from another instructor. On the 46.5-point decomposed section, mean absolute difference was 1.26 points per exam, the observed mean difference was -0.28 points, and 84% of item scores matched exactly. The separate 25-point holistic essay had mean absolute difference 3.88 points. These sample-bounded results do not estimate a general accuracy rate or isolate the effect of rubric decomposition. A June 2026 test-retest study selected 10 final-exam submissions by deterministic stratification across version, batch, and score range and graded each three times. All 10 retained the same letter grade; mean within-exam total-score standard deviation was 0.91 points. A separate earlier 30-exam, three-run capture observed 1.03 points; because it used a different corpus and rubric, it is context rather than a formal comparator. The project page at travisfraser.com/rubrica is maintained together with this report. Revised record (2026-09-05). Reclassified the documentation-only deposit as a report; narrowed the privacy statement; replaced determinism with measured test-retest stability; corrected the cross-model method, sample-size guidance, and human-reference wording; and documented the third validation study.