How well do LLMs understand code?
TL;DR
SemBench is introduced, a novel benchmark consisting of 1000 diverse C programs sourced from the CodeParrot GitHub-code dataset, with 15,404 semantic questions spanning six basic but fundamental properties: dead code-statement, data dependency, function reachability, dominator, dead code-loop, and liveness.
Abstract
Large language models have performed impressively in code generation tasks, yet it remains unclear whether they understand code semantics and whether this affects their ability to write high-quality code. To address this question, we introduce SemBench, a novel benchmark consisting of 1000 diverse C programs sourced from the CodeParrot GitHub-code dataset, with 15,404 semantic questions spanning six basic but fundamental properties: dead code-statement, data dependency, function reachability, dominator, dead code-loop, and liveness. We evaluate 16 widely-used models across 7 families. Even the best model reaches only 80.42% accuracy, and failure rates range from 19.58% to 86.01%, showing a substantial gap between code generation and static semantic understanding. Performance varies sharply across semantic categories. Function reachability shows the strongest correlation with HumanEval and MBPP among the tested categories, suggesting that specific static semantic abilities may partly explain code-generation success. Overall, our experiments underscore the substantial gap between the static semantic understanding and code completion capabilities of modern LLMs.