When Comments Mislead: An Empirical Study of Outdated Comments in LLM-Based Code Completion

Abstract

Large language models (LLMs) are widely used for code completion, where natural-language comments serve as key contextual signals. While prior work shows that comments and natural-language descriptions can improve generation quality by providing high-level semantic guidance, their reliability in evolving codebases remains unclear. In practice, comments may be outdated or inconsistent with the current implementation, raising the question of whether such context may mislead generation rather than simply degrade it. In this paper, we present an empirical study of outdated comments in LLM-based code completion. We find that outdated comments tend to introduce a systematic misleading effect: instead of merely degrading performance, they can bias generated code toward historical implementations, producing outputs that remain globally plausible while failing in regions where the implementation has changed, as outdated comments often retain partially correct yet inconsistent information. Importantly, this effect is not uniform. Our study shows that the impact of outdated comments varies with factors such as the function signature length and task complexity, reflecting the interaction between structural signals and natural-language guidance. Overall, our findings challenge the common assumption that richer context is uniformly beneficial for code generation. They highlight that natural-language comments can act as strong but unreliable signals, and underscore the need for more robust handling of inconsistent contextual information in LLM-based code generation.