"Classical vs. AI-Based Methods in Autonomous Vehicles - Reproducible Experiments from Perception to Field Safety Data"

Abstract

"AV-Stack-Eval is a reproducible research dataset for comparing classical and AI-based methods across the autonomous vehicle (AV) software stack. It packages the inputs, source code, per-replicate results, summary statistics, tables and figures of twelve controlled experiments (E1\u2013E12). The experiments cover perception, state estimation and mapping, trajectory prediction, planning and decision-making, motion control, federated learning, and field safety evidence, all analysed under one statistical protocol: fixed seeds, held-out test data, validation-based tuning, 95% confidence intervals, and paired Wilcoxon tests with Holm correction.The dataset contains three kinds of input:- a seeded sample of 400 real dashcam frames and 1,000 semantic masks from the public comma10k dataset (MIT licence);- seeded simulations of GNSS\u2013odometry fusion, occupancy mapping, trajectory prediction, grid path planning, lane-change reinforcement learning, and Pure Pursuit, Stanley, LQR and MPC path tracking;- transcribed public safety tables covering 220.6 million rider-only miles from the Waymo Safety Impact Data Hub, together with California DMV testing reports and NHTSA fatality rates.Outputs include per-image, per-seed and per-run raw results (JSON), 15 CSV files reproducing every quantitative table of the companion article, 25 figures including 11 system block diagrams (PNG, PDF, SVG), a data dictionary, a protocol description and SHA-256 checksums.Headline results:- Context features raise drivable-area IoU from 0.701 to 0.803.- Sensor fusion reduces GNSS-outage error from 265.5 m to 30.2 m.- Q-learning lowers lane-change crash rate from 59.8% to 6.3% relative to an equal-information rule.- Validation-tuned controller rankings reverse between 8 and 15 m\/s.- FedAvg recovers 54\u201374% of the benefit of pooled training.- Injury crashes are 83% below human benchmarks (rate ratio 0.170 [0.143, 0.201]), while about 388 million miles would be needed to demonstrate an 80% fatality reduction.All results regenerate with one command on a single CPU core. The dataset supports teaching, benchmarking, method comparison and meta-research on reproducibility in automated driving. "