When Tuning Isn't Enough: Limits of Inference-Time Controls and Collaboration in Small Local LLMs

Abstract

We study the limits of inference-time parameter tuning and multi-model collaboration for locally-deployed small LLMs (3B-14B) in a small on-premises deployment using quantized models on a 16 GB GPU. Although these techniques are widely used in local, compliance-constrained deployments to improve model quality without retraining, their actual effectiveness and failure modes remain under-examined. Across code correctness, NLP classification, and summarization, we sweep temperature, top- $p$, and top- $k$ across single-model baselines and pairwise “context-passing” collaboration. Tuning consistently improves collaborative baselines, yet rarely surpasses the strongest single model on precision-sensitive tasks; summarization is the exception, with modest gains. Penalty terms exhibit negligible effect. Additional benchmark validation on MBPP, SST-2, and XSum confirms the same trend: naive context passing does not reliably outperform the strongest single-model baseline, while self-consistency is most useful for summarization. We present a task-sensitivity taxonomy showing where tuning helps, where it saturates, and why collaboration can degrade accuracy due to context contamination, where one model's output biases or pollutes the next model's reasoning. These results provide practical guidance for on-premises, compliance-sensitive deployment settings and establish reproducible baselines to evaluate small-model tuning and collaboration strategies in future work.