Nov 2026· Journal of computing in civil engineering· 0 citations· 31 references
Abstract
Society is steadily embracing the idea of coinhabitant service robots in the built environment for assistance with household tasks and chores. In particular, the growing number of older adults, individuals with disabilities, and people requiring daily assistance has further created an urgent need for intelligent coinhabitant robots capable of supporting mobility, independence, and safety within physical living spaces. The growing complexity of modern built environments and the increasing prevalence of coinhabitant robots have underscored the need for advanced computational methods to support analysis, monitoring, and management. To provide users with meaningful assistance and clear scene understanding in such settings, robots must be able to perceive, interpret, and reason about their surroundings in a way that aligns with human spatial understanding. Place segmentation, which delineates spatial regions such as rooms or functional areas, together with semantic recognition, which assigns meaningful labels to these regions, forms the basis for accurate scene understanding in building-scale environments. Existing deep learning approaches have largely relied on closed-vocabulary detection systems, which are limited to a predefined set of categories of rooms and therefore struggle to interpret diverse or naturalistic descriptions of spaces. Furthermore, most current methods neglect the uncertainty inherent in scene recognition—for example, when rooms have overlapping functions (e.g., kitchen–dining areas) or when people use linguistically similar descriptions to refer to distinct spaces—thereby reducing their effectiveness in ambiguous or complex environments. To address these challenges, we propose an open-vocabulary semantic segmentation and classification pipeline that leverages vision–language models (VLMs) and large language models. Our framework follows a “segment–detect–select” paradigm for open-vocabulary scene understanding, consisting of three modules: (1) segmentation of distinct rooms and regions in the scene; (2) VLM-based similarity scoring between descriptive queries and segmented regions; and (3) adaptive conformal prediction to select among candidate regions while explicitly accounting for uncertainty. Evaluation results on the large-scale Matterport3D dataset, which comprises diverse photorealistic indoor environments, show that the proposed framework outperforms state-of-the-art open-vocabulary methods in scene segmentation [by 30% in AP50 and 5% in mean intersection over union (mIOU)], classification (by 200% in F1 score and 263% in mean average precision), and selection (by 50% in room mIoU). These advances are significant for built environments, as improved scene understanding directly supports safer, more adaptive, and more intuitive coinhabitant assistive agents for humans in complex indoor spaces.
With the advancement of artificial intelligence and robotics, indoor service robots are gradually being deployed in complex environments including residences and hospitals to perform diverse tasks. However, relying solely on traditional localization and obstacle avoidance capabilities can no longer meet the complex demands of real-world scenarios. Robots must possess semantic-level understanding of environmental objects, spatial relationships, and service objectives. Currently, environmental representation for indoor robots is undergoing a transition from being geometry-dominated to semantic-enhanced. This study centers on scene perception and task semantic understanding for indoor service robots. It aims to explore how robots build a systematic cognition of scenes, objects, user commands and task workflows based on low-level visual and spatial data. Using literature review and case analysis methods, this paper synthesizes representative achievements in fields such as semantic mapping, semantic navigation, semantic SLAM, explicit knowledge representation, and task planning. The study argues that scene perception and task semantic understanding constitute a continuous intelligent chain from "environment recognition" to "task execution". In the future, indoor service robots require further improvements in multimodal information fusion, knowledge-driven modeling and task reasoning for open scenarios, so as to enhance their practical performance and operational reliability.
Haoxiang Huang· Applied and Computational En...· 0 citations
This framework combines simultaneous localization and mapping (SLAM), visual-language feature extraction, incremental semantic and instance label fusion, and spatial graph construction to enable a construction robot navigation framework that supports open-vocabulary language queries.
Charles M. Raines, I. Fernandez, Mandy Sun et al.· Journal of computing in civi...· 0 citations
Unmanned aerial vehicles (UAVs) are used for a wide range of tasks, many of which are autonomous. Since UAVs operate in three-dimensional space, visual data presents a rich source of input for them. Meanwhile, the rapid development of artificial intelligence has given rise to vision-language models (VLMs). These models not only recognize objects in a scene, as traditional computer vision techniques do, but also demonstrate a strong understanding of scene context, enabled by large language models (LLMs) serving as one of their core components. However, applications as specialized as aerial robotics typically require custom-trained models, which demand significant computational resources, large datasets, and expensive hardware. Motivated by these limitations, we investigate whether a state-of-the-art, general-purpose VLM can be leveraged for UAV-relevant tasks through carefully designed prompts, without additional training. We evaluate the VLM’s understanding of aerial scenes through the task of image captioning, using visual question answering that targets spatial information within the scene. Furthermore, we utilize VLMs to automatically generate metadata for aerial datasets, enabling more complex future tasks. Results show that VLMs can generate accurate metadata with high agreement with human-annotated data even without retraining. They also demonstrate that careful prompt engineering enhances the model’s ability to reason about and attend to spatial information, highlighting its potential in aerial robotics applications.
Elena Wachtler, Antonella Barisic Kulas, Tamara Petrović et al.· International Conference on...· 0 citations
Abstract. Accurate semantic segmentation of 3D point clouds is essential for applications in photogrammetry, robotics, and large-scale scene understanding. While recent point-based architectures such as PointNeXt achieve strong performance through hierarchical feature learning, they still struggle near semantic boundaries, where points from different classes share local neighborhoods and feature aggregation leads to oversmoothing and ambiguous predictions. To address this limitation, we propose a lightweight boundary-aware learning framework that explicitly models boundary regions during training. The method introduces an auxiliary boundary prediction head that learns boundary cues from local semantic disagreement and integrates them into the segmentation process through a simple late-stage feature fusion mechanism. This design enhances feature discrimination near class transitions without modifying the backbone architecture or increasing inference complexity. Experiments on the S3DIS benchmark with the standard 6-fold cross-validation protocol show consistent improvements over the PointNeXt baseline, achieving gains of 3.22% in mean Intersection over Union (mIoU) and 2.85% in mean class accuracy (mACC) (relative), with notably improved predictions along object boundaries. These results show that incorporating boundary-aware supervision provides an effective and efficient approach to improving segmentation quality in challenging regions.
Waseem Iqbal, J. Paffenholz· The International Archives o...· 0 citations
3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI. Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding. We present GaussDet, a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities. We learn instance features for individual Gaussians to decompose the scene into 3D instance groups. By rendering these groups and aggregating semantic votes from multi-view 2D detections, we generate a robust View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a strong regularizer, attenuating spurious labels caused by low-quality instance grouping. Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding. Extensive evaluations across two key tasks -- open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF) -- demonstrate that GaussDet achieves consistent improvements over existing methods. Most notably, we achieve a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting.
Jameel Hassan, Y. Ranasinghe, Vishal M. Patel· arXiv.org· 0 citations
Point cloud-based understanding has become an important enabler for facility operation and maintenance involving indoor building components. However, existing methods output only discrete labels without explaining component functions or natural language interactions. This paper proposes Building-MLLM, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks. Building-MLLM addresses semantic concentration through four domain-specific mechanisms: Point Information Enhancer for task-relevant semantics, Geometry-Preserving Regularization preventing geometric erosion, fixed textual prefix for domain stabilization, and multi-dimensional LoRA balancing recognition with reasoning. A multi-constraint progressive instruction-generation engine is developed to compile a synthetic point cloud-text dataset with 4198 objects, 37,782 instruction-following pairs, and 47 categories. Experiments show that Building-MLLM achieves 88.00%, 65.10%, and 68.14% on the three task types, respectively, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.
Shuju Jing, Chao Yin· Automation in Construction· 0 citations
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
