Swing the Otter: An Evidence-Normalized Oscillation Audit for Generative AI Systems

Abstract

Swing the Otter is a proposed black-box behavioral audit for generative AI systems. It evaluates whether a system’s semantic position on a fixed proposition changes in proportion to changes in independently verifiable evidence. The protocol separates source-derived evidence, user assertions, and model self-report; includes neutral and leading-pressure conditions, fresh-session replication, and retrieval-manifest controls; and introduces diagnostic constructs including Evidence-Normalized Oscillation (ENO) and Unsupported Flip Count (UFC). The paper explicitly treats these constructs as proposed methods requiring empirical calibration rather than validated universal metrics.