PolyComp, a procedurally generated and verified benchmark that stresses visual recognition and compositional spatial reasoning, is introduced, and the observed accuracy spread across geometry families is larger than across presentation formats.
Abstract
We introduce PolyComp, a procedurally generated and verified benchmark that stresses visual recognition and compositional spatial reasoning. In each problem, a model must identify which of four options shows a pair of polycube components that can be combined to form a target solid. The benchmark contains 120 problems across four geometry families, and each problem has three different presentation formats using either a single image or multiple images. The random guessing baseline is 25%. Across the three presentations (360 presented problems per model), GPT-5.6 Sol with max effort attains 50.0% accuracy (95% problem-cluster CI 43.3-56.7%) at a mean cost of \$0.951 per presented problem, Claude Fable 5 with max effort attains 39.4% (33.1-46.1%) at \$0.701, and Gemini 3.1 Pro Preview with thinking level high attains 27.5% (22.8-32.5%), near the 25% random guessing baseline, at \$0.350. The observed accuracy spread across geometry families is larger than across presentation formats. We present a problem development and evaluation protocol, cost and token accounting, and release the 120 problems.
Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce \textit{\ours{}}, a benchmark with tab-and-blank interlocking pieces where geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (4$\times$4 to 16$\times$16), we find that \textbf{zero-shot VLMs largely lack geometric reasoning}: only one of five frontier models (GPT-5.5) exceeds random baseline on 4$\times$4 puzzles, while all others perform at chance level. While supervised fine-tuning achieves $>$97\% on 4$\times$4, \textbf{all models collapse on larger grids}: GPT-5.5 drops from 70\% to near-random on 8$\times$8, and even fine-tuned models fall below 5\% on 12$\times$12. This ``scaling cliff''suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. \ours{} establishes scalable geometric reasoning as an open challenge for vision-language models.
Shawn Li, Wei Yang, Jike Zhong et al.· 0 citations
This work proposes the Disentangled Spatial Reasoner (DiSR), a simple yet effective framework that reconstructs the physical world into structured 3D evidence using off-the-shelf expert perception models and fine-tunes an LLM with LoRA to perform reasoning solely over this explicit geometric evidence.
Haoze Sun, Jie-Quan Cui, Qingshan Xu et al.· 0 citations
Multi-Trace Synthesis (MTS) is proposed, which converts each symbolic program into heterogeneous reasoning traces, including executable Python scripts and CoT-augmented variants, and Perception-Augmented training, which parses diagrams into structured semantic clauses before deduction, and Consensus-Guided Multi-Trace Ensemble for efficient self-adaptive inference.
Xiaoqiang Kang, Shengen Wu, Maizhen Ning et al.· 0 citations
We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.
Mary Lynn Martin, Yifei Zhang, M. Palmer et al.· 0 citations
OmniMech is introduced, the first million-scale benchmark for evaluating VLMs on executable CAD generation from industrial manufacturing data, and experiments show that current VLMs and CAD-specialized models still struggle with executable program synthesis, fine-grained 3D reconstruction, and reliable enforcement of dimensions and tolerances.
Taiting Lu, Runze Liu, Ziwei Dong et al.· 0 citations
Vision-language model (VLM) evaluation on rendered 3D stimuli is a computer graphics stimulus-design problem: camera, visible faces, silhouettes, shading, and answer format together determine what evidence the model receives. We introduce a progressive surface-disclosure protocol that reveals mesh faces at matched surface-area budgets, renders them as texture-free views, and queries the VLM under category-choice, object-choice, or free-response tasks. We evaluate it on Core-100, a controlled set of 100 everyday 3D mesh models, with five local VLMs. Disclosure ordering produces large threshold differences: in the multi-view object-choice setting, random face disclosure reaches a censored mean threshold of 28.5%, while principal-axis sweep and spatially contiguous growth require 45.7% and 46.4%. The direction repeats across all five models and survives a projected-coverage adjustment and a random-growth ablation that removes the large-face seed prior. Because the forced-choice first-hit metric accumulates chance, we pair the censored thresholds, which size the effect, with a label-shuffle chance-corrected comparison, which confirms it is above chance. At the 10% budget random disclosure beats its shuffle baseline by 28.7 points while connected barely does, so only spatially distributed orderings clear chance at low budgets. Task format matters too: object-choice succeeds on 85.1% of sequences, while strict free-response naming succeeds on 56.8% (68.2% with a fixed alias table). Silhouette-only and coverage-matched experiments rule out interior shading and coverage magnitude, narrowing the driver to the distributed projected shape; a contrastive CLIP baseline shows a much weaker same-direction effect, suggesting the sensitivity is amplified in generative VLMs. The results position progressive rendered 3D VLM evaluation as a measurement protocol whose reports should include disclosure distribution and connectivity, view and task policy, scoring rules, and censoring choices.
Heeyoung Park, Jinyoung Choi, Youngjin Park· IEEE Access· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
