Agentic AI Automation for Data Science in Large-Scale Projects: Intelligent Agents and Loop Engineering
Automating data science on large-scale projects has become a problem of engineering loops rather than tuning models by hand. This paper takes a design stance on that problem and presents LEAP, a loop-engineered agentic pipeline in which an orchestrator agent coordinates stage agents through three feedback loops: a fast loop that searches model configurations under a compute budget, a medium loop in which a critic verifies candidates and revises the pipeline, and a slow loop that monitors and retrains under drift. We make two parts of the design precise. We show that a blind search needs on the order of 1/p trials to find a configuration that is within tolerance of optimal, which motivates cheap multi-fidelity screening, and we show that the optimism of selecting the best of N configurations by validation grows like √2 ln N, which motivates a cross-validated critic. We validate the design with a fully reproducible experiment written from scratch, and we report honest results. At equal compute, agentic multi-fidelity search reaches a test AUC of 0.749 against random search’s 0.738 and wins in 79 percent of runs; a critic loop improves test AUC at every search intensity and curbs the validation optimism that heavy search injects, which rises to 0.037 under the heaviest search; and the critic advantage holds as dimensionality grows. Crucially, a naive greedy agent does not beat random search, which is a strong baseline. The value of agentic automation therefore lies in how it allocates compute and how it selects, not in the promise of a cleverer model, and both can be engineered and reasoned about.