Learning to Assign Prediction Tasks to Agents with Capacity Constraints
TL;DR
This work provides a general theoretical characterization of the problem of learning to assign prediction tasks to one agent from a set of available agents, including human decision-makers and AI models and develops a framework of sequential explore-exploit policy-learning algorithms that seek to maximize overall performance.
Abstract
We address the problem of learning to assign prediction tasks to one agent from a set of available agents, including human decision-makers and AI models. We focus on sequential learning of agent expertise and assignment policies where each agent is constrained to handle a fraction of tasks. We provide a general theoretical characterization of this problem in terms of agent capacities, differences in agent expertise, and task context. We then develop a framework of sequential explore-exploit policy-learning algorithms that seek to maximize overall performance. Experimental results over a variety of tabular, image, and text prediction tasks demonstrate systematic gains from our policy-learning algorithms relative to non-contextual baselines across different types of agents, including LLMs and humans.
