DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making

Abstract

A financial agent can repeat a decision while changing the work behind it. DFAH-Bench operationalizes the Determinism--Faithfulness Assurance Harness (DFAH), pairing decision agreement with tool-path agreement on the same qualified replays, then extends that qualification principle to evidence, authorization, execution and task outcomes. Retrospective and prospective replay analyses expose process variation behind stable decisions. Across 570 eligible prospective episodes, decision agreement is 94.2-95.1%, while agreement on ordered tools, arguments and results is 45.0-51.5%; one stratum falls one group below its prespecified coverage minimum. A separate capture diagnostic shows that systematic omissions can preserve perfect replay agreement. Using the $\tau$-Knowledge banking environment, we retain 1,080 scheduled episodes and 1,033 known native outcomes across separate cohorts with open-weight and frontier generators. Missing outcomes prevented the planned tests, so comparisons are descriptive. On the primary schedule, structural checks alone yield more successes than either gate-and-recovery bundle. The typed-choice bundle has lower mean episode cost than the generative bundle on complete task pairs, but produces fewer successes under every assignment of unknown outcomes. Input limits and recovery behavior materially shape these results. Fixed-state probes reveal higher decision agreement alongside lower agreement with constructed policy labels, and separately expose sensitivity to retained generator rationale in a selected authorization case. Together, the findings connect replay observability to evidence, authorization, completion and cost: evidence sufficiency needs direct assessment alongside repeatability.