Benchmarking Large Language Models on HDL Generation of Cryptographic Primitives

Abstract

While large language models (LLMs) are now competent at generating software in languages such as C and Python, their use to generate hardware description language (HDL) for digital circuits is less explored. In this paper we evaluate seven LLMs on the generation of Verilog for 25 crypto-graphic primitive tasks drawn from LFSR-based constructions, AES, and PRESENT. The results are bimodal: models correctly interpret semi-complex operations such as GF (28) finite-field multiplication and matrix-vector multiplication directly from a mathematical description (pass@1 above 0.9 in several cases), but fail almost entirely on compositional tasks that connect multiple submodules and on tasks that require moderately large bodies of text (no model exceeds pass@1 ≈ 0.05 on the AES S-box, AES round function, or AES top module). We also synthesise correct completions and find that LLM-generated modules can match or beat hand-written reference area, indicating that despite their limitations the models explore diverse and usable implementation strategies.