Rubrica v0.1.5: software description and validation report

Abstract

This report documents Rubrica v0.1.5, a local application for grading scanned, handwritten exams against an instructor's rubric. Under the default local-Ollama configuration, cover-page extraction runs on the instructor's machine. Stored names, SIDs, and page 0 are excluded from external grading requests; those requests contain an anonymous identifier, answer-page images, and rubric text. Rubrica does not inspect answer pages or rubrics for incidental identifiers, so instructors should review and remove that information before external grading. Version 0.1.5 is the current pilot build (September 2026), a signed and notarized standalone macOS application for Apple Silicon Macs on macOS 12 or later (Tauri shell around the Flask backend, React interface). Its grading and scoring logic is unchanged from version 0.1.2, the first pilot build distributed to outside testers in late August 2026. The validation evidence below is carried forward from the previous version of this record; the studies were run on development builds between April and July 2026, before the 0.1.2 pilot build, while parts of the grading prompt and its safeguards were still changing. This report contains a software description, a validation note, architecture and grading-pipeline diagrams, a README, and citation metadata. It does not contain source code or an executable. The deposited documents and diagrams are released under CC BY 4.0. That license does not cover the absent AGPL core source or proprietary pilot application. Validation evidence. An author-run, matched-input cross-model comparison (April 2026; 30 exams, 840 scored items; Claude Sonnet 4.6 vs. o4-mini) reported ICC(3,1) 0.964, quadratic weighted kappa 0.883 on letter grades, 96.2% within-one-point agreement, and mean absolute error of 0.15 points per question. The observed production-minus-reference mean difference was +0.64 points per exam; its approximate 95% confidence interval spans zero. The comparison estimates cross-model concordance and serves as a structural check, not a third-party audit or human-accuracy study. A July 2026 comparison against human-assigned reference scores covered one 25-exam macroeconomics midterm from another instructor. On the 46.5-point decomposed section, mean absolute difference was 1.26 points per exam, the observed mean difference was -0.28 points, and 84% of item scores matched exactly. The separate 25-point holistic essay had mean absolute difference 3.88 points. These sample-bounded results do not estimate a general accuracy rate or isolate the effect of rubric decomposition. A June 2026 test-retest study selected 10 final-exam submissions by deterministic stratification across version, batch, and score range and graded each three times. All 10 retained the same letter grade; mean within-exam total-score standard deviation was 0.91 points. A separate earlier 30-exam, three-run capture observed 1.03 points; because it used a different corpus and rubric, it is context rather than a formal comparator. The project page at travisfraser.com/rubrica is maintained together with this report. Version 0.1.5 record (2026-09-14). Updated for the 0.1.5 build: the packaged app runs on macOS 12 or later; update checks and first-time sign-in now verify TLS certificates against a bundled certificate store (builds 0.1.3 and 0.1.4 could not complete those connections), and each release is tested for this inside the shipped application before signing; review cards show status as a color wash with the status in words. It also corrects the comparison-model label: the author-run comparison can use Google Gemini 3.1 Pro (the in-app default), Anthropic Claude Opus 4.6, or OpenAI GPT-4o, and the April 2026 study used OpenAI o4-mini (the previous version listed Opus 4.7, which Rubrica uses only to re-check escalated multiple-choice answers). Other changes: in-app comparison metrics no longer withhold QWK and ICC below 30 comparisons, one heading in the rubric auto-enhance prompt was reworded, and the packaged app embeds the python.org build of Python 3.14. Grading and scoring logic and the privacy boundary are unchanged. Earlier revision (2026-09-05). Reclassified the documentation-only deposit as a report; narrowed the privacy statement; replaced determinism with measured test-retest stability; corrected the cross-model method, sample-size guidance, and human-reference wording; and documented the third validation study.