Skip to content
Book Open access

A Shared Task on Agentic AI in University Administration

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 29 references

Abstract

Agentic AI systems combine large language models, information retrieval, planning, and tool use to act on behalf of users in complex digital environments. Evaluating such systems under laboratory conditions remains challenging because real-world environments are hard to control: executing AI agents in them is neither safe nor reproducible. We show that university administration provides a particularly suitable setting for Cranfield-style evaluations of AI agents in a laboratory. By reconstructing the digital environment available to university staff, we enable the systematic evaluation of interactive AI agents within human administrative workflows. TIRA's sandboxed execution environment allows submitted agents to operate safely on institutional data. Focusing on selected workflows, such as travel reimbursement, we organize the first shared task on agentic AI in university administration and contribute a reusable benchmark with expert-judged reference solutions. The shared task comprises two subtasks: in agentic retrieval, an agent only finds the documents relevant to a task instance; in agentic task solving, the agent also performs the task and produces the solution.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.