Spatiotemporal vision transformers with Byzantine-robust federated prompt tuning for continuous urban perception

Abstract

Introduction While spatial Vision Transformers (ViTs) achieve high precision in urban scene parsing, their frame-by-frame application in autonomous driving suffers from severe temporal flickering and prohibitive retraining costs across decentralized vehicle fleets. Methods To overcome these dual bottlenecks, this paper introduces a unified Spatiotemporal Hierarchical Mask-Refinement (ST-HMR) framework integrated with a Byzantine-Robust Federated Learning (BR-FL) protocol. The ST-HMR module caches fine-grained prompt tokens via an asymmetrical Temporal Cross- Attention buffer to enforce inter-frame geometric continuity. Concurrently, the BR-FL pipeline employs Multi-Krum geometric distance filtration to aggregate 3 localized prompt gradients from decentralized fleets securely, updating only a 1.4% active parameter subset. Results Evaluated on the Cityscapes Video dataset, the ST-HMR framework improves the video segmentation mean Intersection over Union (mIoU) to 83.5%, elevates the Temporal Consistency (TC) score to 88.5, and reduces depth Absolute Relative Error (Abs Rel) to 0.085, all while maintaining real-time edge processing at 38 FPS. Under severe adversarial network conditions (up to 30% Byzantine/malicious sensor nodes), the BR-FL protocol achieves a 98.4% Byzantine detection rate and maintains a global mIoU of 81.9%, while reducing Over-The-Air (OTA) transmission payloads by over 99% (3.8 MB vs. 1.2 GB per round). Discussion These findings demonstrate that parameter-efficient prompt caching eliminates temporal boundary jitter without heavy 3D transformer overhead, while geometric gradient filtering provides robust defense against decentralized poisoning, establishing a scalable, secure, and temporally coherent perception paradigm for next-generation edge robotics.