SDBench is introduced, a generic framework for generating evaluation datasets for domain-specific LLMs that reduces the reliance on expert man-power while ensuring that the collected data is uniformly distributed.
This study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures, and will open source \phi-Bench to facilitate public research on using LLMs for fs development.
ESF-Bench is introduced, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots over 8 unique domains and curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments.
Toby Liang, Gopal Sarda, Sagar Davasam et al.· 0 citations
Providing accurate and efficient user support with the emerging large language model (LLM) techniques for high-performance computing (HPC) centers is extremely challenging. The major obstacle lies in the integration of specialized knowledge that is not available in the base LLMs. Supervised Fine-tuning (SFT), Retrieval Augmented Generation (RAG), and In-Context Learning (ICL), are the common techniques to empower LLMs with domain knowledge. However, there is a lack of comprehensive understanding of their question-answering capabilities in the HPC domain. In this work, we present a comparative study of eight candidate methods by combining the base model (Llama-3.1-8B-Instruct), SFT, RAG, and ICL using a ticket database of 60,000 question-answer pairs from Texas Advanced Computing Center (TACC). We evaluate these methods with five metrics: correctness, coherence, relevance, fluency, and cosine similarity, using both human and LLMs as judges. Our results show that ICL+RAG with the base model achieves the highest correctness in general, and ICL with the base model outperforms others for debugging and testing questions. The results suggest that combining ICL, RAG, and a powerful base model is a promising solution for LLM-enabled user support in computing centers.
Mingkai Zheng, Fangru Linghu, Sikan Li et al.· Practice and Experience in A...· 0 citations
The findings show that, while LLMs achieve promising results, they struggle with harder problems and with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.
Rodrigo Pato Nogueira, Marco Vieira, João R. Campos· Empirical Software Engineeri...· 1 citation
In this work we introduce a novel application of Large Language Models (LLMs) towards the classification of malicious Windows Portable Executable (WinPE) files. Rather than fine-tuning an LLM based on the raw binary of the samples serving as the corpus, in this case we leverage our in depth knowledge of the underlying structure of the WinPE format to apply a two phased approach to translate each file into human readable format. For our studies we focus on the header portion of the files in order to reduce the required time for the fine-tuning process, performing Parameter Efficient Fine Tuning (PEFT) applying a QLoRA adapter to a light-weight baseline model. In addition, we deploy our fine-tuned model according to Software Defined Networking (SDN) principles. The model is containerized, with constraints applied in terms of the amount of CPU, memory, and networking resources that are available. In addition, the model is deployed as a virtualized LLM (vLLM) adhering to a server-client architecture. We benchmark the fine-tuned model against the SoReL-20M dataset and compare results in terms of both accuracy, including the macro-averaged F1 score, and latency, including the Time-to-First-Token (TTFT). By fine-tuning the model to understand the language of the Windows Portable Executable format, we demonstrate a significant improvement in macro-averaged F1 score, up to an absolute difference of 50 percentage points over generic LLMs, while maintaining comparable latency.
Timothy Miskell, Yan Luo, Peilong Li et al.· International Conference on...· 0 citations
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
