LLMs help robots understand vague instructions and focus on key details
To help robots do chores in places like homes and factories, a new approach from MIT uses one language model to clarify users’ instructions, then another to ignore irrelevant info.
More from the blog
How mobility gives language models a deeper understanding of place
Algorithms & Theory
Empty shelves or lost keys? Recall is the bottleneck for parametric factuality
Generative AI
With a feel for physics, AI models simulate a wider range of real-world scenarios
“GeoPT” helps AI models understand the basics of physics so they can simulate how objects respond to things like wind and water more efficiently and accurately.
Solving the solvent problem
By focusing on electrolytes, MIT scientists are making sodium-metal batteries a more practical energy storage option.
Related papers
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.
A Multiagent Large Language Model–Based System for Early-Stage Building Layout Planning
A multiagent large language model (LLM)–based system for early-stage building layout planning, which enables flexible design requirement inputs and robust spatial reasoning and demonstrated significant improvements in both geometric quality and semantic alignment over a baseline LLM-only system.
Neurosymbolic Embodied Agents
A neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning and evaluates executable continuations using a domain-independent planning heuristic is presented.
Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis
This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis, and shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space.