STACK: 3D Spatial Task Analysis in Collaborative Construction Keyframes
TL;DR
STACK is presented, to the knowledge the first benchmark pairing real multi-party construction video with dense 3D annotations and goal-paired judgment, exposing a gap between perception and judgment in current VLMs and informing their use as analyzers of collaborative group dynamics.
Abstract
Analyzing multi-party physical collaboration means tracking what a group builds and whether their actions move the shared structure toward a goal. Vision-Language Models (VLMs) could automate this analysis from session recordings, but existing spatial-reasoning benchmarks use rendered scenes or curated snapshots, not real collaborative video. We present STACK, to our knowledge the first benchmark pairing real multi-party construction video with dense 3D annotations and goal-paired judgment. Using 237 keyframes from 19 LEGO sessions, we evaluate seven VLMs across input modality, structural complexity, and manipulation judgment. A structured JSON state drives reconstruction accuracy to near-perfect across all models. Rendered views outperform real video frames, while additional viewpoints give only marginal gains. For manipulation judgment, frontier models reach 80–92% accuracy while open 7–8B models stay at chance. A key dissociation emerges. Small open models identify what manipulation happened with structured context but cannot judge whether the result is spatially correct. These findings expose a gap between perception and judgment in current VLMs and inform their use as analyzers of collaborative group dynamics.