Structured Experience Replay with Objective-Driven Dense Rewards for Mapless Mobile Robot Navigation Using Deep Q-Networks

Abstract

Autonomous mobile robot navigation in unknown environments remains challenging due to sparse reward signals, unstable value estimation, and action-selection imbalance in Deep Q-Networks (DQN). This paper presents a DQN-based mapless navigation framework that combines objective-driven dense reward shaping with a structured experience replay mechanism to improve learning stability and policy robustness. The proposed reward function provides continuous guidance through goal-oriented heading alignment, LiDAR-based safety–progress coupling, and a time-step penalty, while retaining terminal success and collision rewards. To address action imbalance, experiences are explicitly organized into multiple replay buffers based on executed actions and reward polarity, enabling balanced mini-batch sampling. The framework is implemented on a TurtleBot3 Burger and evaluated in ROS–Gazebo simulation and real-world indoor environments. Compared to a standard uniform replay buffer under identical network and reward configurations, the structured replay mechanism improves the highest average reward by 97.2% and increases the navigation success rate from 40% to 85%. Real-world experiments achieve a 100% success rate across 30 trials, demonstrating robust sim-to-real transfer and reliable autonomous navigation performance.