Language-Guided Generation of Specific Driving Scenarios Using Diffusion Models

Abstract

. Scenario-based simulation has now become essential for validating autonomous driving vehicles, and real world trials alone cannot enable the validation of rare but critical-to-safety situations. However, simulation-based validation is effective when the simulated scenarios that are generated are realistic, diverse and controllable. Simulated scenarios from scripted methods are very controllable but have poor scalability; fully data-driven methods fail to be semantic in nature. Recent advances that combine natural language interfaces with diffusion-based generative models give a promising and complementary approach. The paper reviews representative methods at this intersection, discusses their system designs and evaluation schemes, and also the open challenges pertaining to controllability, executability and reproducibility and particularly focuses on scene-level conditional diffusion models with language. This review summarizes the current research trends and provides guidelines for achieving more controllable, executable and repeatable scenario-generation pipelines in the future autonomous driving validation framework.