Intelligent Autopilot Agent based on Neural Network Technologies in an Autonomous Driving Simulation Environment

Abstract

The paper presents the architecture of an intelligent agent for end-to-end autonomous vehicle control based on the M3Drive neural network and multimodal data fusion in the CARLA simulation environment. The proposed approach combines visual features (depth maps and panoptic segmentation for reconstructing 3D geometry and scene semantics), navigation commands and spatial coordinates of route control points, as well as time series of agent states (history of control actions and critical event indicators). The model architecture includes specialized encoders for each data type: EfficientNet-lite0 for visual modalities followed by cross-attention fusion, bidirectional LSTM with an attention mechanism for processing temporal context, and a gating module for tabular features. The combined representations are fed into a multilayer perceptron with residual blocks and multi-task output "heads" that predict continuous control actions: steering angle, acceleration, and braking. Training was conducted in two stages with a weighted loss function. In the second stage, the model was fine-tuned using scenarios involving turns and reactions to traffic lights, which reduced steering errors by more than half. Experimental results at the independent test site confirmed the effectiveness of the system: a route completion rate (RC) of 89 % and an integral driving score (DS) of 67.6 were achieved at a high performance of 58 FPS. The results confirm the applicability of the proposed architecture for creating lightweight real-time autonomous driving systems and lay the foundation for transfer to the Mitsubishi i-MiEV physical platform.