TraceUX: An Explainable Rule-Based Framework for Context-Aware Static UX Evaluation

A Context-Aware Platform for Evaluating LLM-Generated Backend Applications

Large Language Models (LLMs) are now widely used for code generation, yet even syntactically correct output may contain logical and semantic errors that remain invisible until runtime, particularly in framework-driven applications, where correctness depends on dependency injection, framework conventions, configuration, library compatibility, and database interaction. This paper presents GenTest, an open-source platform for context-aware dynamic generation of Java code using LLMs, runtime compilation and class loading, Spring bean registration, JPA/PostgreSQL execution, and declarative YAML-based test validation. GenTest supports multiple LLM providers through a provider-agnostic interface and evaluates generated components within a live Spring Boot application context. Across 209 test cases and 933 assertions, GenTest achieves a 63.9% assertion pass rate, and 94.2% of failures occur after successful compilation, confirming that execution-centered evaluation reveals failure modes invisible to static analysis.

From Trace to Line: An Empirical Study of What Drives LLM-Based OSS Vulnerability Localization

Large language models (LLMs) show promise for vulnerability discovery, yet prevailing methods often inspect code in isolation, struggle with long project contexts, and produce coarse function- or file-level predictions that offer limited guidance for targeted patches. This paper presents a controlled empirical study of what factors help LLM agents perform project-level, line-level vulnerability localization from runtime failure evidence. We introduce T2L (Trace-to-Line), a reproducible research framework that narrows repository-scale code into candidate vulnerable lines through AST-based chunking, structured diagnostic information collection, and evidence-guided refinement. To support systematic evaluation, we build T2L-ARVO, an expert-verified 50-case benchmark spanning five crash families in real-world open-source projects, and T2L-Agent, a modular reference agent for studying localization workflows under equal per-case budgets. Across ten models and controlled ablations, we find that structured diagnostic analysis, coarse-to-fine refinement, and multi-hypothesis exploration all improve trace-to-line localization. Across these workflow-level configurations, the largest observed increase in line-level localization reaches 48.9 percentage points over the corresponding baseline, while increasing reasoning budget or adjusting sampling temperature yields limited and inconsistent benefits. Overall, our study provides an empirical reference for designing LLM-agent workflows for complex project-level localization tasks that move from runtime failure evidence to precise vulnerable-line hypotheses.

Engineering Practices in Open-Source Agent Frameworks: A Large-Scale Empirical Study

A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.

ReATest: enhancing policy-as-code workflows through automated test case generation from Rego policies

TAG: A Lightweight Framework for Test-Driven Agentic Artifact Generation

Generating structured artifacts with Large Language Models - e.g.\ database queries, threat framework mappings, entity schemas - is relatively straightforward; however, making them reliable enough for production deployments presents challenges. We present TAG, a lightweight framework based on a core principle: \textit{LLMs generate, we validate}. This reframing shifts responsibility from generation quality to validation rigor. The framework rests on three key attributes: First, \textbf{test driven generation}: when tests fail, the LLM receives indicative error messages that expose why the output failed, enabling the LLM to understand its mistakes and refine subsequent attempts. Second, \textbf{deterministic and LLM-based tests}: deterministic tests catch heuristics that can be programmatically verified (schema, syntax, cross-reference), while LLM-based tests evaluate nuanced semantic and delicate features that resist programmatic inspection (intent alignment, logical consistency, domain correctness). Third, \textbf{expert-distilled judges}: LLM-based tests are calibrated to distill and replicate human expert decision distribution, transforming manual human quality gates into scalable, reusable evaluation proxies that reflect professional-grade validation standards. We demonstrate the framework on three artifact types in the security domain - KQL query generation, MITRE ATT\&CK mapping, and entity mapping - deployed in production at Microsoft Sentinel. We believe this framework can be applied beyond security to other artifact generation tasks, providing a path to reliable, high-quality outputs without sacrificing the efficiency gains of LLM generation.

Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects

The integration of AI-powered coding agents into Continuous Integration/Continuous Delivery (CI/CD) pipelines has fundamentally altered how software verification is conducted. While these agents successfully automate the test generation, current evaluation benchmarks (e.g., SWE-bench) largely focus on pass-rates rather than the intrinsic quality of the generated tests. This raises the possibility of “stealth technical debt”, in which test suites pass execution but do not offer comprehensive coverage or semantic value. We address this methodological gap through a large-scale, empirical comparison of 204,673 test artifacts which comprises of 24,941 human-authored files and 179,732 agent-generated files; sourced from the AIDev dataset. Using the Abstract Syntax Tree (AST) parsing with Python's naive ast module, we implemented a “white-box” static analysis framework to evaluate three quality dimensions: Assertion Strength (RQ1), Edge-Case Coverage (RQ2), and Flakiness Potential (RQ3). Our results present a nuanced inversion of traditional assumptions. AI agents performed better than humans in Edge-Case Coverage, with almost twice the variety of boundary checks (Variety Score: 0.62 vs 0.32) and a higher frequency of null-safety testing (13.40% vs. 8.3%), even though human developers had a slight advantage in Assertion Strength (88.1% strong assertions vs. 85.37% for agents). But this thoroughness comes at a price: due mostly to their reliance on file I/O and non-deterministic logic, agent-generated tests exhibited a higher risk of flakiness (Candidate Rate: 0.41 vs. 0.30). These findings suggest that while AI agents excel at rigorous boundary testing, they lack the “environmental awareness” needed to write stable, hermetic tests.