Parameter-efficient fine-tuning is widely assumed to improve small open-weight models on the specific behaviors it targets. We present a controlled case study that complicates this assumption for multi-tool agent execution. Using Qwen2.5-7B-Instruct and a fixed 23-task, machine-checkable benchmark, we show that an initial 35-trajectory QLoRA adapter (v1) regresses from the base model's 22/23 (95.7%) to 18/23 (78.3%)—despite the training data being disjoint from the benchmark at the prompt level. Trace-level inspection of every failure attributes the regression to two distinct causes: a serving-layer tool-call parsing defect, and insufficient coverage of chained and error-handling tool-use patterns in the training data. Correcting the parser and adding 11 trajectories targeting the observed failure categories yields v2, which passes all 23 tasks. We treat this recovery cautiously: because the augmentation categories were selected by inspecting which benchmark categories failed, the result is better characterized as closing an identified, benchmark-specific gap than as evidence of improved general tool-use ability, and we report a category-level train/eval information leakage threat that this design introduces. Using the recorded failure traces, we further separate, to the extent the evidence allows, which recovered failures are attributable to the parser fix versus the data augmentation. We release the evaluation harness, training data, serving code, and full result artifacts to support replication and critique.
Alyssa L· Zenodo (CERN European Organi...· 0 citations
Parameter-efficient fine-tuning is widely assumed to improve small open-weight models on the specific behaviors it targets. We present a controlled case study that complicates this assumption for multi-tool agent execution. Using Qwen2.5-7B-Instruct and a fixed 23-task, machine-checkable benchmark, we show that an initial 35-trajectory QLoRA adapter (v1) regresses from the base model's 22/23 (95.7%) to 18/23 (78.3%)—despite the training data being disjoint from the benchmark at the prompt level. Trace-level inspection of every failure attributes the regression to two distinct causes: a serving-layer tool-call parsing defect, and insufficient coverage of chained and error-handling tool-use patterns in the training data. Correcting the parser and adding 11 trajectories targeting the observed failure categories yields v2, which passes all 23 tasks. We treat this recovery cautiously: because the augmentation categories were selected by inspecting which benchmark categories failed, the result is better characterized as closing an identified, benchmark-specific gap than as evidence of improved general tool-use ability, and we report a category-level train/eval information leakage threat that this design introduces. Using the recorded failure traces, we further separate, to the extent the evidence allows, which recovered failures are attributable to the parser fix versus the data augmentation. We release the evaluation harness, training data, serving code, and full result artifacts to support replication and critique.
Alyssa L· Zenodo (CERN European Organi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
