Home Lab: evaluation pre-registration

Abstract

This project evaluates a self-hosted AI agent lab running on a single Mac mini (Apple M6, 32 GB, macOS 27.0). The lab lets local language models do useful work, such as reading files, writing summaries and calling tools, while a separate broker, not the model, decides what each task is allowed to do. The questions are practical. Can a small typed routing model beat a fixed rubric without promoting weak evidence (H1)? Does grammar-constrained decoding reduce the tool calls a strict parser has to refuse (H2), and does putting the tool schema in the prompt do the same (H2b, secondary)? Is a tuning change adopted only when it shows a measured gain (H3)? Do the lab's isolation controls still stop prompt-injection attacks when a real model, rather than a scripted worst-case stub, is driving the agent (H4)? The main model is Qwen3-Coder-30B-A3B at 4-bit (MLX and GGUF builds, pinned by commit and file hash). All runs use temperature 0 and a fixed seed, and each run writes a sealed record of the exact code commit, model revision, task file hash and machine settings. Rates are reported with 95 percent bootstrap intervals. Some real-model runs were made before this registration. Amendment 1 lists every one of them and treats them as exploratory. The confirmatory H2 and H2b tests use a new held-out set of 70 tool-call tasks that no model had seen at registration time, and H4 is a replication at the registered commit. Expected outcomes: a clear yes or no for each hypothesis under the stated decision rules, published whichever way it falls, and a set of measured numbers (memory, speed, tool-call reliability, injection resistance) for running agents on a 32 GB machine. Full plan, hypotheses, decision rules and frozen file hashes: docs/PREREGISTRATION.md at github.com/roshanaryal1/home-lab, commit d8726b43be5290afbb2ce6ae2f6722632705d9a8.