Skip to content
Open access

The effect of syntactic and semantic information on word grounding through visual perception

Sep 2026 · Frontiers in Artificial Intelligence · Vol 9 · 0 citations · 33 references
Medicine

TL;DR

This study quantitatively evaluates how syntactic and semantic information affects word-grounding performance and demonstrates the contribution of linguistic structure and semantic representations to perceptual alignment.

Abstract

Introduction Word grounding refers to the ability of agents to associate linguistic terms with the perceptual concepts they represent, such as color, shape, and spatial features. This study quantitatively evaluates how syntactic and semantic information affects word-grounding performance. Methods We evaluated five configurations within a generative multimodal Bayesian framework. The baseline grounds words using perceptual cues, while the additional configurations independently incorporate word-position indices, part-of-speech tags, FastText embeddings, and transformer-based contextual embeddings. The models were evaluated using a CLEVR-derived 3D scene dataset and synonym-substituted descriptions. Results Linguistic information improved grounding accuracy across the Color, Geometry, and Spatial Relation modalities. Contextual embeddings produced a 53.8% absolute improvement in object-grounding accuracy over the baseline. Only models incorporating semantic embeddings generalized effectively to synonym-substituted descriptions without retraining. Discussion These findings demonstrate the contribution of linguistic structure and semantic representations to perceptual alignment and provide empirical foundations for developing language-grounding systems.

Read PDF

Similar papers

Linguistic Features for Interpretable Textual Entailment

SLITE is presented, an explainable hybrid model for Recognizing Textual Entailment that integrates two complementary layers of semantic analysis: a structural-relational layer, based on semantic compatibility and incompatibility between compositional entities, and a distributional-informational layer, based on structur...

David Torres-Moreno, J. Hermosillo-Valadez, Asela Reig-Alamillo · 0 citations
Conference Open access Sep 2026

Syntactic Structure-Guided Visual Grounding with Subject-Centric Feature Enhancement and Verification

This work designs a semantic structure-based feature refinement module to adaptively modulate subject-centric and contextual visual features and incorporates a visual semantic consistency verification module, which leverages a subject-aware contrastive learning strategy to constrain and verify the semantic corresponden...

Jie-Peng Cai, Zhen Xu, Tie-Song Zhao et al. · 0 citations
Open access 2026

Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

A layer-wise analysis indicates that surface-level features such as temporality and negation are captured more reliably than deeper semantic phenomena like quantification in large language models, highlighting the limited capacity of current LLMs to generate fully formal meaning representations.

Rémi De Vergnette, Maxime Amblard · 0 citations
Open access 2026

Byte-pair Encoding Captures Aspects of Word Meaning that Distributional Semantics Overlooks

Large language models learn word meanings through distributional semantics (i.e., patterns in text), and distribution does not always capture information about taxonomy and function, which are essential to human representations of word meaning. However, sound and spelling often conveys such information, and language...

David A. Haslett, Antoni B. Chan, J. Hsiao · 0 citations
Open access Aug 2026

From Word Embeddings to Semantic Projections: Interpretability and Context in Web-Scale Semantic Analysis

This paper revisits semantic projections and related count-based representations as interpretable directional semantic structures for semantic analysis in document corpora and web-based information environments and demonstrates that semantic projections effectively capture persistent contextual structures while remaini...

Mabel López-Bordao, Antonia Ferrer-Sapena, Pablo Lara-Navarra et al. · 0 citations
Open access Aug 2026

Minimal symbol grounding through language statistics: An information theoretic approach quantifying uncertainty

One of the fundamental issues in the cognitive sciences concerns the question of how language attains meaning. The current study investigated how the extent to which language statistics and symbol grounding reduce uncertainty about word meaning can be quantified. Using Shannon’s information theory, conditional entropy...

M. Louwerse · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.