The first systematic empirical study of defects introduced during this stage of deep learning compilers in TorchDynamo, the default DLC frontend for PyTorch 2, the most popular DL framework is conducted, using a domain-knowledge-enhanced LLM-aided methodology.
Abstract
Deep learning compilers (DLCs) are designed to translate deep learning programs into optimized, hardware-specific code. Typically, DLC frontends translate programs into graph-based intermediate representations (IRs) to enable optimizations. Defects introduced during this stage (termed \emph{fBug}s) are severe yet understudied, as prior work predominantly focuses on low-level APIs and operators or treats DLCs as monolithic entities. To bridge this gap, we conduct the first systematic empirical study of \emph{fBug}s in TorchDynamo, the default DLC frontend for PyTorch 2, the most popular DL framework. Leveraging a domain-knowledge-enhanced LLM-aided methodology, we analyze 123 \emph{fBug}s and construct a taxonomy comprising 7 root cause categories and 15 subcategories. Our findings provide actionable insights for DLC development and testing. Furthermore, we leverage the LLM to generate targeted, root cause-aware test cases to detect new bugs. We uncovered 23 previously unknown \emph{fBug}s in recent releases (15 confirmed) across eight (sub)categories, demonstrating the efficacy of our methodology in testing and hardening DLC frontends.
Deep learning (DL) compilers such as Apache TVM translate high-level models into optimized low-level code through multi-stage compilation pipelines. While recent testing efforts have improved fuzzing of optimization stages, they still face two key challenges: (i) the lack of semantics-preserving test models, leading to low validity, and (ii) coarse-grained input generation that fails to trigger hard-to-reach compiler components. To address these limitations, we propose CovCraft, a unified testing framework that integrates constraint-driven model generation with large language model (LLM)-guided input adaptation. CovCraft constructs diverse and valid ONNX models via symbolic constraint encoding and SMT solving, and then iteratively refines inputs using LLM-guided prompts to target uncovered functions, enabling the activation of rarely executed code paths. We evaluate CovCraft on TVM and observe consistent improvements over state-of-the-art techniques: it increases branch and function coverage by 8.9% and 7.0%, respectively, and detects 8 bugs. Moreover, the LLM-guided component achieves an 83.75% success rate in covering designated target functions, demonstrating the effectiveness of combining constraint-based generation with adaptive LLM reasoning for DL compiler testing. The prototype implementation of CovCraft is publicly available at: https://github.com/duduhedangdang/CovCraft.
Yifei He, Fangyu Yang, Ting Su et al.· Annual International Compute...· 0 citations
The next iteration of Nova is presented, an automated end-to-end JIT compiler that achieves absolute control over hardware mapping by synthesizing fine-grained kernels directly from the computation's structure by extending Nova's compilation pipeline to natively support full Transformer architectures.
Adwaid Suresh, Aparna A. Harshini, Jona Delcy et al.· 0 citations
VistaFuzz is introduced, a document-guided fuzzing technique that uses a locally served open-sourced LLM to extract parameter specifications from API documents and generate inputs that satisfy both parameter constraints and inter-parameter dependencies.
Bin Duan, Tarek Mahmud, Meiru Che et al.· 0 citations
: Large language models are increasingly being deployed in safety-critical domains, yet remain vulnerable to jailbreak attacks that circumvent safety alignments. This systematic review synthesizes empirical jailbreak research published between 2024 and 2025, using a PRISMA-guided search protocol, followed by BERTopic-based topic modeling. The analysis identifies eight main jailbreak categories: optimization-based, ge-netic/evolutionary, iterative refinement, semantic/persuasion-based, decomposition, context/generation-level, visual/encoding and fuzzing attacks, and characterizes their effectiveness, efficiency, and transferability across open-source and proprietary models, including Llama-2/3, Vicuna, GPT-3.5/4, Claude, Gemini, and DeepSeek-V3. Results show that simple configuration and context-level attacks can match the near-perfect attack success rates of sophisticated white-box optimization methods on models such as Llama-2, while requiring far fewer queries and no parameter access, highlighting a gap between research focus and practical threat severity. The review further identifies five recurring vulnerability mechanisms: representation-level gaps, execution-priority manipulation, semantic fragmentation, gradient-space exploitation and persuasion susceptibility, and documents family-specific vulnerability patterns, with open-source Llama-based models consistently more exposed than safety-enhanced architectures such as Claude. Diverse methods, uneven focus on models and publication bias limit how broadly results apply. Nonetheless, the review reveals that weaknesses in safety alignment persist across successive LLM generations, urging that effective defenses must address all eight attack categories rather than isolated techniques.
Meda Račaitytė, Hélder Bastos, R. Ribeiro et al.· International Conference on...· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
LLM-based code generation is now embedded in mission-critical pipelines, but defenses against vulnerable output remain post-hoc -- static analyzers, fine-tuned classifiers, or an LLM judge that screen completed code, ignoring the generating model's own internal state. We test a narrower, directly measurable question: when an LLM reads a piece of C/C++ code as context, do its hidden activations already carry a signal about that code's vulnerability status? We extract last prefill token activations from four LLMs (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) across three model families and train MLP probes on these activations. We evaluate them on four function-level C/C++ benchmarks (Devign, Big-Vul, Draper VDISC, PrimeVul). Our probes achieve 41.7\% average F1 using 13.4--16.0M-parameter probes -- under 0.2\% of base-model size. On Devign, the best probe (Qwen3.5-9B, 68.8\% F1) matches the published fine-tuned-classifier SOTA (67.9\%) despite reading only a frozen, general-purpose LLM's activations; on the harder, more imbalanced benchmarks (Big-Vul, Draper VDISC, PrimeVul) probes trail SOTA substantially. This is early evidence that a coding LLM's own representation of arbitrary code is informative about that code's vulnerability status, motivating further work toward lightweight, model-native vulnerability screening.
Alizishaan Khatri· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
