ASAREE: An Analytical Sandbox for Agentic AI Research, Engineering, and Experimentation
TL;DR
This work uses ASAREE to evaluate key design choices in a mutli-agent machine learning pipeline and highlights ASAREE as a robust framework for evaluating agentic system performance and resource efficiency.
Abstract
Summary Agentic AI platforms enable the engineering of autonomous workflows but are not designed for experimentation and hypothesis testing. ASAREE (Analytical Sandbox for Agentic AI Research, Engineering, and Experimentation), is an open-source platform to address this gap. ASAREE creates agents, connects to MCP servers and tools, and designs factorial experiments through a visual interface or Python SDK. It records a full provenance trace for every run and routes all model calls through a provider-agnostic bridge that supports local deployments, ensuring data privacy. As a use-case, we use ASAREE to evaluate key design choices in a mutli-agent machine learning pipeline. Across a 2 × 2 × 2 factorial design, more advanced models, greater reasoning effort, and critic agent use significantly increased compute time, token use, cost, and feature count without improving predictive performance. The lowest-cost baseline, Claude Sonnet 5 with medium effort and no critic, achieved the highest mean PR AUC while Claude Opus 5 with extra high effort and a critic agent cost 15.5× more (USD) and ran 13.1× longer while performing worse on average. These findings highlight ASAREE as a robust framework for evaluating agentic system performance and resource efficiency. Availability and implementation ASAREE is available on GitHub at: https://github.com/EpistasisLab/ASAREE. Contact jason.moore@csmc.edu Supplementary Information Supplementary information is available at https://github.com/EpistasisLab/ASAREE/tree/main/publications/bioinformatics