2026· Proceedings of the 15th International Conference on Data Science, Technology and Applications· 0 citations· 33 references
TL;DR
A framework for scalable, dataset-scale LLM-based data transformation that processes data row-by-row in batches over lazily evaluated, distributed dataframes, enabling LLM-based transformations on arbitrarily large datasets without loading them into memory.
Abstract
: Applying large language models (LLMs) to real-world tabular datasets at scale is subject to a fundamental mismatch between model context limits and enterprise data size. Existing approaches address this by truncating data or fitting everything into a single prompt — methods that do not scale and sacrifice row-level semantic understanding. We present a framework for scalable, dataset-scale LLM-based data transformation. It processes data row-by-row in batches over lazily evaluated, distributed dataframes, enabling LLM-based transformations on arbitrarily large datasets without loading them into memory. Three composable primitives cover the most common patterns: MAP for field-level enrichment, FILTER for row selection, and REDUCE for semantic deduplication via embedding-based similarity search. An agentic orchestration layer accepts a plain-language goal, plans the required primitive sequence, and routes operations that require no semantic reasoning to generated Python code. We evaluate the framework on five domain-specific benchmarks. On a 51,291-row agentic enrichment task, the pipeline achieves 96.2% accuracy in 9.4 minutes. On a 3-million-row deduplication task, REDUCE completes in 1.90 hours. Single-primitive benchmarks reach 99.6% coverage on category extraction, Macro F1 of 0.97 on sentiment classification, and 96% accuracy on compliance filtering. The framework is released as open source at https://github.com/vitalops/datatune.
Large language models (LLMs) and LLM-based agents are increasingly being deployed to automate complex workflows, promising to revolutionize data management and processing. However, existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored. To bridge this gap, we introduce DataClawEval, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. Built upon production-grade code authored by professional enterprise data engineers, it comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts. Evaluating 16 frontier agents exposes critical limitations: The strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine, revealing strict domain specialization rather than omnipotent proficiency. Thus, autonomous data engineering remains a formidable, unresolved challenge. We release our dataset, containerized environments, and deterministic evaluation scripts at https://github.com/Dicemy/DataClawEval/tree/master
Debin Meng, Jiaming Yang, Zefang Zong et al.· 0 citations
Large language models (LLMs) excel at analyzing individual documents but break down on exhaustive, cross-entity analytical questions over enterprise-scale datasets due to context overflow, loss of per-entity attribution, and linear latency from sequential tool calls. We present BatchDAG, a system in which an LLM generates a typed directed acyclic graph (DAG) of operations -- SQL queries, semantic searches, in-memory transforms, parallel fan-outs, and single-shot analyses -- which a deterministic engine evaluates with topological-wave parallelism and structured JSON data flow. A key optimization, entity-aware batching, groups rows by logical entity before fan-out, reducing LLM calls by up to 47x. BatchDAG is not primarily an accuracy improvement over hand-optimized pipelines; rather, it is a general-purpose orchestration layer that replaces multiple hand-engineered workflows with a single system that generates the appropriate execution strategy from natural language. In controlled experiments on 12 transcript-heavy queries, BatchDAG (3.74/5) achieves quality comparable to an expert-designed pipeline (3.25/5) and significantly outperforms a ReAct agent (3.09/5, p<0.01), with superior provenance (77% transcript evidence rate vs. 46-60% for baselines). A controlled ablation shows structured JSON intermediates reduce hallucinations by 27% versus prose summaries (paired t-test, p=0.107, n=12). The planner achieves 98.8% valid-DAG rate across 300 planning calls. In production at Brevian.ai, BatchDAG processes queries over 50,000+ meetings in under 60 seconds, with measured per-query costs of $0.02-$0.24 at published GPT-5.1 pricing.
BIRD is a recent enterprise-scale Text-to-SQL benchmark that exposes a regime not covered by earlier datasets such as Spider: it consists of 95 real-world databases, which on average contain seven tables each, with abbreviated column names, dense foreign key graphs, and noisy domain values. Generation-target intermediate representations (IRs) such as SemQL and NatSQL, which reduce the model’s output space to a structured plan and rely on a deterministic compiler to reconstruct full SQL, were developed before BIRD and evaluated mainly on Spider-like schemas. As a result, most subsequent BIRD-era work on Text-to-SQL has shifted toward prompt and pipeline engineering, while leaving the choice of generation target—raw SQL versus IR—largely unexamined for large, messy enterprise databases. In this work, we revisit generation-target IRs under the BIRD regime and propose HyMIR, a typed JSON IR designed for enterprise-scale schemas. HyMIR removes high-error structural clauses—JOIN, GROUP BY, and the SELECT projection—from the model’s output and delegates their reconstruction to a compiler that infers join paths from the foreign key graph, while a predicate passthrough channel carries the long tail of fragments that resist abstraction, including LIKE predicates, range bounds, nested IN subqueries, and dialect-specific date functions. To isolate the effect of the IR itself, we compare HyMIR against standard SQL as the generation target under strictly matched conditions: identical training data, backbone (Qwen-2.5-Coder-7B-Instruct), fine-tuning recipe (DoRA), and sampling protocol. On the BIRD development set, HyMIR improves execution accuracy from 52.50% to 55.48% under raw decoding and from 56.81% to 58.44% with $N{=}11$ result-set self-consistency. Across model scales (1.5B, 3B, 7B), the gains exhibit a mid-scale sweet spot—largest at 3B, with improvements of roughly + 11 points, and smaller but consistently positive at 7B—such that a 3B HyMIR model approaches the performance of a 7B SQL-tuned model while using fewer than half the parameters. Finally, a compiler-only oracle round-trip, which encodes gold SQL into HyMIR and compiles back, achieves 92.83% execution accuracy on BIRD and 91.0% on Spider, indicating comparable representational coverage across the two benchmarks and quantifying a hard upper bound on attainable model performance under this IR and compiler stack.
LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users'perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.
Xiaoyu Zhang, Xianyun Cheng, Tianlin Li et al.· 0 citations
Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts. We call this disconnect the \textit{NL2Pipeline gap}. To bridge it, we introduce \textsc{DataFlow-Harness}, a platform that guides an LLM agent to construct platform-native directed acyclic graphs (DAGs) through typed, incremental mutations rather than free-form scripts. The platform combines \textsc{DataFlow-Skills} for procedural guidance, a Model Context Protocol (MCP) layer that exposes the live operator registry and current pipeline state, and \textsc{DataFlow-WebUI}, which synchronizes conversational authoring with a visual DAG editor. On a 12-task data-engineering benchmark, \textsc{DataFlow-Harness} achieves a 93.3\% observed end-to-end pass rate. Relative to Vanilla Claude Code, it reduces measured monetary cost by 72.5\% and generation latency by 49.9\%; its observed pass rate is within 0.9 percentage points of the Context-Aware Claude Code baseline while its cost is 42.8\% lower. Per-task analysis indicates that Skills are most useful when construction depends on implicit procedural knowledge. These results show that live platform grounding can produce persistent, editable workflow artifacts with an observed reliability close to script-generation baselines and with lower measured construction cost and latency.
Runming He, Z. Wong, Hao Liang et al.· 0 citations