Artificial IntelligenceNatural Language Processing
Abstract
Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation.
Investigating how experienced developers use agents in building software, including their motivations, strategies, task suitability, and sentiments finds that while experienced developers value agents as a productivity boost, they retain their agency in software design and implementation out of insistence on fundamental software quality attributes.
This systematic review evaluates 55 studies from 2017 to 2023 on the application of machine learning techniques to ASD, highlighting key challenges and opportunities, particularly the need for models that can integrate complex data to improve diagnostic accuracy and treatment outcomes.
Rafael Muñoz-Terol, Jesús Peral, Sandra Amador et al.· Heliyon· 4 citations· ⚡1
SimulRAG, a simulator-based RAG framework with a generalized retrieval interface that translates between text and simulator parameters/outputs, is proposed, which improves informativeness and factuality over the strongest adapted RAG baselines, while UE+SBA enhances claim-level efficiency and quality.
Haozhou Xu, D. Wu, M. Chinazzi et al.· arXiv.org· 3 citations
This paper identifies two different routes through which models can acquire geometrically separable features: they can learn them from complementary co-occurrence signals in general language data, including text-number co-occurrence and cross-number interaction, or from multi-token addition problems.
A novel quantum generative model for synthesizing tabular data by proposing a quantum generative adversarial network architecture with flexible data encoding and a novel quantum circuit ansatz for effectively modeling tabular data is introduced.
P. Bhardwaj, Caitlin Jones, Lasse Dierich et al.· Scientific Reports· 2 citations
New MIT research could lead to better materials for a fossil-fuel-free process for making the chemical that's essential to fertilizer and other products.
