Revisiting TuRTLe: A Comprehensive Evaluation of LLMs for RTL Generation

Benchmarking LLMs for Verilog Design Flows

A reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs, enabling reproducible evaluation of generative AI for hardware design workflows.

Code Transformation Rule Synthesis using LLMs: Potential and Limits

Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. To mitigate this, we conduct a novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep. We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software- evolution tasks: API misuse correction, program repair, API migra- tion, and language version migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicability. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.

Route-Align-Verify for Functional Correctness in Code Generation

The results indicate that functional correctness in code generation can be meaningfully improved without modifying the backbone architecture, by jointly optimizing how tasks are prompted, how the model is adapted, and how final outputs are selected.

Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations

Large language models (LLMs) are increasingly being explored for automating SystemVerilog Assertion (SVA) generation, yet most evaluations report correctness on a single syntactic representation of an input. Such point accuracy does not reveal whether a model's correct output is stable when the same RTL behavior is written differently. This paper presents a controlled metamorphic evaluation of LLM-based SVA generation under semantics-preserving RTL transformations. Starting from the VERT dataset, we construct a quality-filtered conditional-control pool and a stratified 40-program evaluation set containing 295 assignment behaviors. We evaluate two open code models, Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite, with an identical evaluation prompt and greedy decoding. Three transformations are studied: operand reordering, deterministic identifier renaming, and redundant parenthesization. Beyond baseline and transformed accuracy, we measure conditional robustness, invariance failure, and any-flip rate, with 10,000-sample clustered bootstrap intervals at the RTL-program level. Across all six model-transformation conditions, 9.7%-27.0% of behaviors that were correct on the original RTL become incorrect after a semantics-preserving transformation. Aggregate accuracy can therefore hide substantial instability: under identifier renaming, DeepSeek-Coder-V2-Lite improves from 53.9% to 63.7% accuracy while 19.5% of its originally correct behaviors fail. Manual review of 30 sampled correct-to-wrong transitions identifies dropped path predicates, branch-polarity errors, Boolean-structure corruption, and output-contract violations. The results show that point accuracy alone is insufficient for characterizing LLM reliability in assertion generation and motivate robustness-aware evaluation for AI-assisted hardware verification.

From Unfinished to Done: Bridging the MDE Implementation Gap with Constrained LLMs

Model-driven engineering (MDE) excels at ensuring architectural consistency and managing complexity, yet extending generated code skeletons with project-specific business logic often remains a time-consuming manual task. Conversely, large language models (LLMs) offer immense flexibility in coding but may produce vibe-coded results—software that looks correct but fails to adhere to intended architectures and implicit requirements. An open challenge lies in harnessing the adaptability of LLMs to handle implementation details without sacrificing the deterministic correctness of the MDE scaffolding. Previous attempts have largely relied on unconstrained coding assistants or required a human in the loop for interacting with the LLM. To overcome this, we introduce Agentic Gap Filling, a hybrid pipeline where LLM prompts are embedded directly within the model to drive the generation of code skeletons. The generated code contains protected TODO regions, which LLM agents subsequently resolve and a jury of LLM review agents and separate testing agents immediately validate. We integrate this approach into the MontiThings framework, demonstrating a workflow where models define the architecture of a project while artificial intelligence (AI) agents act as developers and reviewers, filling in the logic. This synergy ensures the structural integrity of the generated system while providing the flexibility of LLMs to adapt to various domain requirements. Our evaluation across 32 test configurations demonstrates that this multiagent workflow achieves up to 93.21% functional accuracy after four iterations, while preserving the structural integrity of the generated system.

Automated Generation of RISC-V Extensions with Formal Correctness Guarantees