This study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures, and will open source \phi-Bench to facilitate public research on using LLMs for fs development.
Abstract
Large Language Models (LLMs) are fundamentally transforming computer system research and development. As we employ LLMs in file system (fs) development, it is essential to understand their capabilities, limitations, and operational efficiency for domain-specific tasks. We present \phi-Bench, an LLM benchmarking framework for fs-specific tasks. To facilitate benchmarking, we develop six types of tasks in \phi-Bench: basic understanding, basic implementation, performance modeling, debugging, optimization, and new feature development. Each type emphasizes different LLM capabilities: instruction following, knowledge recall, reasoning, or coding. To create high-quality tasks while achieving broad coverage with minimal human effort, we develop a new AI-assisted task generation pipeline in addition to expert-written and textbook-adapted tasks. With 505 tasks in \phi-Bench, we conduct an empirical study with both open source (DeepSeek-V4-Flash, GLM-5.1, and MiniMax-M2.7) and proprietary (Claude-Opus-4.7, GPT-5.2, and Gemini-3.1-Pro) LLMs. Our study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures. We will open source \phi-Bench to facilitate public research on using LLMs for fs development.
The findings show that, while LLMs achieve promising results, they struggle with harder problems and with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.
Rodrigo Pato Nogueira, Marco Vieira, João R. Campos· Empirical Software Engineeri...· 1 citation
SDBench is introduced, a generic framework for generating evaluation datasets for domain-specific LLMs that reduces the reliance on expert man-power while ensuring that the collected data is uniformly distributed.
Cheng Guo, Kai Hu, Shuxian Liang et al.· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
Providing accurate and efficient user support with the emerging large language model (LLM) techniques for high-performance computing (HPC) centers is extremely challenging. The major obstacle lies in the integration of specialized knowledge that is not available in the base LLMs. Supervised Fine-tuning (SFT), Retrieval Augmented Generation (RAG), and In-Context Learning (ICL), are the common techniques to empower LLMs with domain knowledge. However, there is a lack of comprehensive understanding of their question-answering capabilities in the HPC domain. In this work, we present a comparative study of eight candidate methods by combining the base model (Llama-3.1-8B-Instruct), SFT, RAG, and ICL using a ticket database of 60,000 question-answer pairs from Texas Advanced Computing Center (TACC). We evaluate these methods with five metrics: correctness, coherence, relevance, fluency, and cosine similarity, using both human and LLMs as judges. Our results show that ICL+RAG with the base model achieves the highest correctness in general, and ICL with the base model outperforms others for debugging and testing questions. The results suggest that combining ICL, RAG, and a powerful base model is a promising solution for LLM-enabled user support in computing centers.
Mingkai Zheng, Fangru Linghu, Sikan Li et al.· Practice and Experience in A...· 0 citations
Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. To mitigate this, we conduct a novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep. We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software- evolution tasks: API misuse correction, program repair, API migra- tion, and language version migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicability. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.
Axel Allain, Aymeric Blot, D. Khelladi et al.· 1 citation
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
