Reproducibility in the Era of AI Agents
TL;DR
The implications of this shift in mindset are explored and lessons learned are shared that hopefully illustrate this shift in emphasis with current agentic technology.
Abstract
Reproducibility efforts should yield easy-to-use, well-packaged, and self-contained software artifacts. However, the goal is to enable AI agents, not fellow human users, to reproduce empirical results. Gone are the days where a user engages with code and documentation directly. Increasingly, this task is delegated to agents, and even in cases where user intervention is required, interactions are mediated by agents. Given this shift in the target audience, developers building software artifacts in support of reproducibility need to optimize for the "agent experience" alongside the "developer experience". In modern parlance, this means making software legible to agents. This paper explores the implications of this shift in mindset and attempts to explain what's changed and how we should adapt. Generalizing on my own recent attempts to "agentify" the Anserini IR research toolkit, I share lessons learned that hopefully illustrate this shift in emphasis with current agentic technology.