This paper proposes two complementary artifacts: an Agency Justification Record (AJR) helps teams decide when an agent is warranted over simpler alternatives and an Agentic Delegation Policy (ADP) captures what must be specified for safe and effective development.
Abstract
Agentic AI systems do not just predict or recommend; they plan, maintain state, and act in external environments with varying degrees of autonomy. This changes the requirements engineering problem in a specific and under-addressed way: it introduces what we call the delegated-autonomy boundary -- the set of decisions about what may be delegated to the system, under what graduated authority, with what oversight, and how control is returned. Current practices bury these decisions inside prompts, tool schemas, and runtime policies, even though they are requirements-level commitments. This paper proposes two complementary artifacts. First, an Agency Justification Record (AJR) helps teams decide when an agent is warranted over simpler alternatives. Second, an Agentic Delegation Policy (ADP) captures what must be specified for safe and effective development: purpose, authority, information, coordination, assurance, and evolution. Crucially, authority in the ADP is modelled as graduated, i.e., a tiered structure. We illustrate the framework with two contrasting examples: a safety-critical hospital discharge coordination agent and an automated code review agent.
Deploying agentic AI in regulated contexts requires knowing two things about a deployment: what the system can do—its agency—and how much it acts without human involvement— its autonomy. Though often treated independently, the two are coupled: at higher autonomy, human error correction is less available, so reliable operation requires constraining agency accordingly, and compliance rules reinforce this by mandating human involvement as the consequences of actions grow. Yet no established approach addresses them jointly as a design problem, leaving practitioners without a principled basis for deciding where oversight should sit and how errors can be caught before they propagate. We introduce a two-dimensional design space in which both dimensions are organised into five operational levels, making the coupling explicit and navigable, and we propose six architectural tactics—checkpoints, escalation, multi-agent delegation, tool provisioning, tool fencing, and write staging—for adjusting a deployment’s position within it. We ground the tactics in a public-sector document classification system, tracing a path from manual operation to near-full autonomy under realistic compliance constraints. Together they offer a shared vocabulary for compliance-aware agentic AI design in which responsibility, auditability, and reversibility are explicit design choices rather than retrofitted properties.
As AI systems increasingly exhibit agentic behavior, discussions of autonomy often conflate what systems are technically capable of doing with what they should be permitted to do in practice. This paper introduces a governance framework that explicitly separates Allowed Autonomy Levels (AAL), which define the degree of autonomy an AI agent is authorized to exercise given risk, oversight, and accountability considerations, from Autonomous Capability Levels (ACL), which characterize an agent's inherent technical abilities. We present a structured set of autonomy levels spanning reactive execution, decision support, supervised action, goal-directed autonomy, and delegated operational authority, and describe how control, reversibility, and accountability change as autonomy increases. To operationalize this framework, we propose a risk-aware decision process for assigning allowed autonomy, analyze how risk and accountability evolve across autonomy levels, and demonstrate its application through a deployed enterprise data engineering agent, illustrating how a system assessed at a high capability level can be deliberately constrained to a lower allowed autonomy based on risk, reversibility, and organizational readiness. By distinguishing authorization from capability, this work provides practical guidance for the design, deployment, and governance of Agentic AI systems.
Hai-Tao Zheng, Qian Dong, R. K. Depena et al.· arXiv.org· 0 citations
This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems, and asks which parts of an AI institution produce safety and how they do it.
Charting these challenges provides a roadmap toward trustworthy autonomous agent deployment: security must become a verifiable property of the architectures, protocols, and runtimes that govern agent behavior, rather than an optional layer of guidance.
Firms are increasingly confronting a fundamental organizational choice: whether to retain human control over operational and marketing decisions or to delegate decision authority to agentic artificial intelligence systems. While recent advances in generative and autonomous AI enable real-time pricing, inventory allocation, and demand coordination, firms exhibit substantial heterogeneity in how much autonomy they grant these systems—ranging from full automation to extensive human oversight. This raises a central operations management question: when should firms delegate pricing and inventory decisions to agentic AI, and how should such delegation be governed? We develop an analytical model of AI delegation at the operations–marketing interface in which a firm jointly determines pricing and inventory under demand uncertainty and chooses among human control, full AI autonomy, or human-in-the-loop governance. Agentic AI improves responsiveness by enabling state-contingent decisions, but also introduces new forms of operational exposure by reducing buffers and accelerating execution. Our analysis yields several key insights. First, we identify a demand-variance threshold above which delegating decisions to agentic AI becomes optimal, even when AI is imperfect. Second, we show that partial delegation can strictly dominate both full autonomy and full human control, providing a theoretical foundation for hybrid governance structures widely observed in practice. Third, when AI investment is endogenous, adoption and autonomy become distinct decisions, generating a three-region equilibrium in which firms may invest in AI while deliberately restricting its authority. We further show that learning, service-level asymmetry, stochastic lead time, endogenous human oversight, and organizational scale fundamentally reshape delegation incentives, often in counterintuitive ways: faster learning can delay early autonomy; improved pricing coordination can increase inventory imbalance; and larger organizations may rely on autonomy even under moderate uncertainty. Together, our results demonstrate that AI delegation is not a technological inevitability but an economically contingent organizational choice shaped by uncertainty, risk asymmetry, and structural complexity. The paper provides a unified theoretical framework for understanding AI governance in operations and offers guidance for firms navigating the transition toward autonomous decision-making.
Abhishek Srivastava· Production and operations ma...· 0 citations
Embedding an intelligent agent in an existing application creates a persistent coordination problem: users can revise goals and manipulate shared objects while delegated execution continues. We argue that dependable integration requires an explicit correspondence between task-level interaction and application behavior. We introduce Agent-Integrated Software (AIS) as a software pattern combining a conventional core, direct interaction, and a built-in agent, and Intent-Level Interaction Abstraction (IIA) as the task semantics through which users inspect and control delegated work. An open transition-system model relates AIS execution to IIA states and events. Interaction contracts constrain this relation through task bindings, role-specific authority, control transitions, and outcome evidence; continuous assurance maintains scoped claims as their dependencies change. A compact disclosure contract and conditional propositions illustrate why local component validity is insufficient and how selected admission invariants can be separated from planning. Contrasting software domains expose the framework's assumptions and limits. This perspective develops a research agenda spanning application abstraction, development support, controlled execution, quality assessment, and human supervision, with the aim of making agent integration a maintainable software engineering discipline.
