Measuring the Cost of Prompt Structure in LLM Code Generation

Abstract

Structured prompting routinely improves general LLM reasoning, but whether it transfers to code generation remains underexplored even as coding tools add template layers around user requests. We compare three levels of automatically imposed prompt structure, from free-form instructions over a static component template to a pipeline that extracts and enforces an explicit output contract, on DS-1000 and SWE-bench Lite. We find that moderate structure makes code more executable but does not resolve more tasks, strict contract enforcement improves format compliance while substantially reducing functional correctness, and the effect depends on the underlying model. For code generation, prompt structure is a tuning knob rather than a best practice, and output contracts belong in validation rather than generation.