When Should Humans Intervene? Capacity-Aware Oversight for Financial AI Agents
Abstract
Human oversight is widely prescribed for high-impact artificial intelligence, but “human in the loop” is not an operational policy when review capacity is finite and agent activity varies over time. We formulate oversight for financial AI agents as a capacity-constrained allocation problem: for each proposed action, a controller decides whether to permit automated execution or consume scarce human review. Unlike a preliminary version of this study, risk scores are not drawn conditional on the realized outcome: observable features are generated first, harm is a hidden logistic function of those features, and a logistic-regression risk model is fit on a separate 60-day, 14,264-action calibration stream, then frozen and evaluated out-of-sample on a held-out 120-day, 28,174-action test stream across five financial domains. The frozen model is well calibrated out of sample (test Brier score 0.091; predicted/observed decile harm rates agree within 1–2 points). We compare no review, random review, a calibration-frozen threshold (uncapped and capacity-truncated), daily risk ranking, a convex risk-severity heuristic, and a genuine expected-loss policy selecting marginal-benefit-maximizing cases under the model’s own optimality argument. The uncapped threshold captures 40.9% of harm value but breaches the 33-action daily capacity on 49 of 120 days; truncating it recovers feasibility at a coverage cost (36.9%). Under the hard constraint, risk ranking captures 38.9% of harm value; the expected-loss policy raises this to 45.5% (a 17% relative gain) and achieves the lowest total expected cost among all evaluated policies, 10.2% below risk ranking at the same 14.1% review rate, while random allocation captures only 13.2%. A 27-cell factorial sensitivity analysis over reviewer effectiveness, review cost, and score discrimination, plus a sweep over harm prevalence and severity dispersion, shows this advantage is stable throughout. Results are computed outputs of the stated synthetic simulation; harm prevention is evaluated in expectation via the assumed reviewer-effectiveness parameter, not observed human-review outcomes. These results show that effective human oversight is jointly a prediction, allocation, and workforce-capacity problem: calibrated risk scores alone are insufficient, and unconstrained cost optimization can redistribute review away from low-severity domains.