Online Failure Detection for Robot Manipulation with Time-Aware Metrics and Efficient Models

Abstract

Vision-Language-Action (VLA) models have recently achieved strong performance in robot manipulation, with promising generalization across diverse tasks and environments. However, long-horizon executions and unseen scenarios can still lead to failures, highlighting the need for online failure detection to enable timely corrective actions. In this context, detection performance should be evaluated not only by accuracy, but also by the timeliness of failure identification. Existing evaluation protocols primarily rely on metrics such as ROC-AUC and threshold-based accuracy to summarize detection performance. However, these metrics largely overlook the temporal dimension of online failure detection, where identifying a failure earlier can be as important as identifying it correctly. Although accuracy-detection time curves offer a way to visualize the trade-off between accuracy and timeliness, they do not produce a unified scalar score, making method comparisons less direct and potentially inconsistent. To address these limitations, we propose AUTC, a timeaware evaluation metric that explicitly captures the trade-off between detection accuracy and timeliness by integrating detection performance over a range of accuracy levels. AUTC provides a unified criterion for model comparison. We also develop a transformer-based failure detector tailored to sequential failure detection. Experiments on robot manipulation benchmarks show that AUTC better captures temporal differences and provides more discriminative evaluation for early failure detection, while the proposed detector achieves competitive performance with improvements in several settings.