The engineering details of ADEPT are presented, a framework that integrates representative adequacy techniques, including neuron-coverage-based metrics, surprise adequacy, input distribution coverage, boundary coverage, and source- and model-level mutation score, under a consistent execution workflow.
Abstract
Over the past decade, many test adequacy metrics have been proposed for deep learning that characterize test dataset adequacy from different perspectives, e.g., neuron activation behavior, latent feature coverage, decision-boundary exploration, etc. However, these metrics are typically released as independent research prototypes with substantially different installation and preprocessing requirements, execution workflows, and configuration mechanisms. These complications make them quite difficult to reproduce, compare, and adopt in research work and practical deployment alike. In this paper, we present the engineering details of ADEPT, a framework that integrates representative adequacy techniques, including neuron-coverage-based metrics, surprise adequacy, input distribution coverage, boundary coverage, and source- and model-level mutation score, under a consistent execution workflow. ADEPT provides a template-based metric interface with well-defined extension points for integrating new adequacy metrics. Furthermore, it provides YAML-based configuration management, preprocessing-cache reuse, and structured result reporting, making it easy to use in any research and development workflows. ADEPT is designed for researchers and practitioners who wish to reproduce and apply adequacy metrics without spending days or weeks implementing missing tooling or configuring disparate research prototypes. A demo video is available at https://aub.ie/ADEPT_video.
This work uses large language models (LLMs) to automatically generate a dataset of neural networks that can serve as a benchmark for validation, designed to cover diverse layer types and to handle multiple input data types and tasks.
The design of transformer-based Large Language Models (LLMs) is being radically changed through new architectures that are able to overcome scalability limitations of previous designs, including Mixture-of-Experts (MoE), Multi-Head Latent Attention (MLA), and Multi-Token Prediction (MTP). As an open-weighted model released at the end of 2024, which has both state of the art architectural transparency and production scale efficiency, DeepSeeek-V3 represents the ultimate testing ground for investigating these modern technologies. This paper provides a comprehensive analysis of the architectural structure of DeepSeek-V3 based upon information from the DeepSeek-V3 Technical Report, industry benchmarking data and independent latency testing, to demonstrate how various techniques can be used to optimize training while still providing competitive performance in code generation and mathematical reasoning. In addition, latency testing conducted on a Distilled version of DeepSeek-V3, with approximately 14 billion parameters, running on a T4 GPU, reveals that although significant improvements have been made in optimizing latency there remains substantial barriers to deploying these models. Through this context, this research will serve as a reference document for practitioners and researchers who wish to understand current trends and challenges in increasing accessibility to high performance AI models.
Yassine Zouhdi, B. Hdioud· EPJ Web of Conferences· 0 citations
DeepQual-Web is introduced, a single multimodal deep learning framework for comprehensive quality assessment and optimization of web applications that combines Gradient Boosting Regression and Bidirectional Long Short-Term Memory networks to interpret the performance and reliability attributes of execution logs and system metrics.
I. Alharbi· International Conference on...· 0 citations
The significance of this work lies in demonstrating that a deployable, explainable detector can be assembled from compact components, and an edge-type ablation study, a cross-dataset evaluation, and a per-vulnerability analysis are reported to characterize the approach.
Ahmed M. Elalfy, G. Ebrahim, M. B. Mansour· Computers· 0 citations
RecCompl is a comprehensive model compilation system that enables efficient model compilation of industrial scale DLRMs with PT2 and introduces a configuration-based user interface that decouples compilation settings from model code, allowing fine-grained control without intrusive changes.
Changji Li, Shuai Yang, Yuzhen Huang et al.· Proceedings of the 32nd ACM...· 0 citations
Evaluating the quality of test suites for Deep Learning (DL) systems generally relies either on structural coverage criteria (e.g., Neuron Coverage), distribution-aware metrics (e.g., Surprise Adequacy), or mutation testing. While mutation testing offers the most reliable measure of fault-detection, it is often prohibitively expensive, as each mutant configuration may require retraining the model from scratch multiple times to establish detectability. This paper investigates to what degree the computationally cheaper coverage and distribution metrics can serve as reliable proxies for mutation testing. We measured the correlation (Spearman and Pearson) between nine adequacy metrics and the DeepCrime mutation score across four datasets (two classification, two regression) at ten test set size proportions. For classification tasks, all metrics except NeuraL Coverage (NLC) demonstrate moderate-to-strong, statistically significant correlations (Spearman 0.53 to 0.69) with the mutation score. For regression tasks, most metrics yield moderate, statistically significant positive Spearman correlations with growing test sets, though partial correlation analysis suggests these are largely mediated by test set size. Our findings suggest that standard adequacy metrics can nevertheless serve as proxies for mutation testing, particularly in classification DL systems.
Tien Duc Nguyen, Gonzalo Oberreuter, Siegfried Steckenbiller et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
