Aug 2026· Journal of Science Research and Reviews· Vol 3, pp. 130-139· 0 citations· 2 references
TL;DR
This 2026 survey synthesises 39 primary sources selected from an initial corpus of 274 candidate records screened over 2020-2026, organises the field around a five-role pipeline taxonomy (perception, prediction, planning, control, and human-machine interaction), and compares six representative driving MLLMs on accuracy, latency, and parameter footprint.
Abstract
Autonomous driving has progressed from rule-based subsystems and modular perception, prediction, and planning stacks toward unified data-driven architectures, and multimodal large language models (MLLMs) are increasingly proposed as the cognitive substrate of the next generation of highly automated road vehicles. This 2026 survey synthesises 39 primary sources selected from an initial corpus of 274 candidate records screened over 2020-2026, organises the field around a five-role pipeline taxonomy (perception, prediction, planning, control, and human-machine interaction), and compares six representative driving MLLMs (DriveGPT-4, LMDrive, Senna, DriveLM, GPT-4V-AD, and Cosmos-1) on accuracy, latency, and parameter footprint. A benchmark coverage matrix over LingoQA, BDD-X, DriveLM, nuScenes-QA, AutoHallu, and CODA-LM exposes evaluation gaps in prediction and planning. Model behaviour is translated into safety-assurance terms by mapping four MLLM failure-mode families to the functional-safety standard ISO 26262, the Safety of the Intended Functionality standard ISO 21448 (SOTIF), and the autonomous-systems safety-case standard UL 4600. A three-tier vehicle, edge, and cloud deployment topology is described together with the digital-twin and over-the-air update infrastructure that surrounds it. The strongest empirical finding is that Cosmos-1 delivers the best accuracy among models with sub-150 ms latency (76.6 percent mean reasoning accuracy at 480 ms), leaving verifiable safety certification as the single most important open problem for closed-loop deployment. The survey closes with a six-item research agenda spanning sub-100 ms real-time inference, out-of-distribution generalisation, multi-agent intent reasoning, verifiable safety certification, long-tail corner-case coverage, and closed-loop sim-to-real transfer. The article is intended as a reference for automotive system architects, safety engineers, regulators, and machine-learning researchers preparing the next generation of automated driving systems.
Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used to validate Autonomous Driving Systems (ADSs), but it remains a fragmented modular pipeline in which scenario generation, retrieval, modification, ADS execution, and results analysis are performed by separate tools with little interaction. Large Language Model (LLM) agents have shown promise across ADS sub-systems such as perception, planning, and control. However, no prior work covers the whole scenario-based testing pipeline for ADSs with a unified LLM-agent framework. We present PlannerForge, an LLM-agent framework that extends all scenario-based testing stages (from Scenario Generation to ADS Assessment) and adds two further LLM-enhanced stages: ADS Enhancement and ADS Benchmarking. We evaluate PlannerForge with 10 off-the-shelf LLMs across all tasks (Generation, Selection, Modification, Module Routing, Planner Testing, and Enhancement) under 5 prompt conditions. Best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends match commercial APIs on most tasks. Open-source models such as Qwen3.6:35B match commercial APIs on three of the five tasks. Chaining the modules end-to-end retains 83% / 78% of seed queries (commercial / open). It outperforms Scenario Factory 2.0 (Finkeldei et al., 2025) on natural-language generation (193 vs. 144 executable of 200) and realises 92-96% of requested city, road and vehicle attributes. It outperforms BM25 (Robertson and Zaragoza, 2009) at rank 1 selection (92.0% vs. 67.5%) and From-Words-to-Collisions (Gao et al., 2025) on physically valid edits (>=94% vs. 31%). At N=400, cost-tuning lifts planner success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4%, without domain-specific fine-tuning.
Yuan Gao, Sebastian Müller, Mattia Piccinini et al.· 0 citations
Autonomous vehicles (AVs) operate in complex environments where failures are consequential. Sophisticated machine learning models for perception and planning are key to overcoming at least part of that complexity, but their black-box nature complicates validation and verification (V&V). The recent integration of Vision-Language-Action (VLA) models into AVs introduces a unique opportunity: besides generating trajectories, these models produce an explicit Chain-of-Thought (CoT) explaining their underlying rationale. This CoT provides a rich specification to cross-check model outputs and detect inconsistencies that may expose unsafe or unintended behavior. This paper assesses whether CoTs from a recent open driving VLA can support such monitoring. We curate DriveAlignBench, a specialized dataset from NVIDIA's Alpamayo 1.5 VLA for AVs containing 150 CoT-trajectory pairs, which we manually annotate for reliability, trajectory consistency, and safety. Our analysis reveals that 33.3% of CoTs are unreliable. Among reliable CoTs, the generated trajectory is consistent with the CoT in 74% of cases. Leveraging this potential, we propose integrating a CoT-trajectory consistency check into a runtime monitor. The check is nontrivial: CoTs express open-vocabulary, scene-relative driving commitments, while trajectories are low-level ego-motion sequences whose semantics depend on road geometry and motion context. To bridge this gap, we develop a family of automated consistency monitors. Our best monitor, lane-relative F-LLM with GPT-5.5, achieves F1 = 0.75, improving over the strongest raw-waypoint LLM baseline by +0.13 absolute F1 and over a rule-based monitor by +0.38. We release DriveAlignBench, the monitor implementations, and annotation tools at https://github.com/776styjsu/drive-the-thoughts.
Tianwen Yu, Lu Feng, Sebastian G. Elbaum· 0 citations
This study develops a human-centered cognitive-agent framework for understanding how large language models (LLMs) can support human-vehicle teaming in automated driving. Following PRISMA guidelines, we reviewed 1,126 records published between 2021 and 2025 and included 52 studies after screening and full-text assessment. The synthesis identified four recurring capability clusters: perception and awareness, reasoning and decision-making, action and control, and interaction and communication. Across these functions, LLMs show promise for improving semantic scene understanding, explainable decision-making, high-level planning, and bidirectional communication with drivers. However, hallucinations, incomplete physical grounding, non-deterministic reasoning, and latency remain important limitations in safety-critical settings. The findings suggest that LLMs are most effective as high-level cognitive partners integrated with verified task-specific modules rather than as standalone controllers. The proposed framework offers design guidance for safer, more transparent, and collaborative human-vehicle systems.
Jing-Jie Wang, Brandon J. Pitts· Proceedings of the Human Fac...· 0 citations
A cognitive dual-process planning framework that represents planning-relevant scene knowledge in a machine-parsable structured chain-of-thought (S-CoT) schema and shows how explicit scene knowledge can be operationalized through adaptive reasoning and rule-based verification to support high-level VLM planning decisions.
Zhongyao Yang, Haoyu Li, Yuchen Yan et al.· arXiv.org· 0 citations
A latent memory pool is constructed that stores failure cases along with their structure scene representations and expert trajectory labels, and a dedicated Retrieve Model that decouples static road structure and dynamic agent interactions to enable structurally grounded retrieval is designed.
Zebin Xing, Yupeng Zheng, Qiangyu Chen et al.· 0 citations
End-to-end autonomous driving has evolved from camera-to-control regression toward planning-oriented systems that use structured representations, trajectory-level outputs, and increasingly realistic evaluation protocols. This survey reviews this transition across behavior cloning, conditional imitation learning, privileged distillation, BEV and vectorized planning, unified perception-prediction-planning architectures, world-model-based planners, and vision-language-action systems. We argue that the key distinction in modern end-to-end driving is not whether intermediate representations are used, but whether they are learned, supervised, and evaluated to support safe, feasible, and route-compliant planning. To organize the literature, we synthesize existing methods along four axes: input representation, planning output, supervision signal, and evaluation protocol. We further examine the benchmark shift from open-loop trajectory matching to closed-loop simulation, non-reactive real-log evaluation, long-tail testing, and human-preference-aware metrics. Our analysis highlights that architectural progress is difficult to interpret without benchmark-consistent evaluation, and that displacement-based open-loop metrics alone provide limited evidence for safe and human-aligned driving. We conclude with open challenges in uncertainty-aware planning, learner-expert mismatch, runtime safety assurance, language-action grounding, world-model validation, and reproducible benchmarking.
Yanchen Guan, Xing-Chen Liu, Bin Rao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
