Dynamic Potential Field Exploration Framework: An Internal Value Construction Method Based on State Visitation History

Abstract

Effective exploration in sparse-reward environments is one of the important challenges in reinforcement learning. Existing methods typically rely on external reward signals or intrinsic motivation mechanisms derived from rewards to establish behavioral value. However, when environmental feedback is extremely limited, agents may lack sustained and explicit exploration guidance, leading to reduced exploration efficiency.This paper proposes a Dynamic Potential Field Exploration (DPFE) framework. Its core idea is to construct an internal value representation independent of external rewards through the agent's own state visitation history. The framework treats the state space as a dynamically evolving potential field, in which the potential energy of visited states increases monotonically with dwell time, while unvisited states maintain low potential energy. The agent learns by perceiving internal feedback signals generated by potential energy differences, thereby forming an exploration process driven by state history and internal value.Unlike methods that rely on external rewards to estimate value, the method in this paper does not attempt to infer latent value from sparse external reward signals, nor does it attempt to inject external information. Through a unified potential energy accumulation rule, the framework ensures that lower potential energy corresponds to higher exploration priority: repeatedly visited regions become less attractive, while unexplored regions maintain sustained exploration priority.This paper further provides a formal definition of the framework, proves the finite dwell property and the cycle elimination property, and explains that under the premise that the agent executes the framework's design specification, the mechanism can avoid infinite dwelling in explored regions and maintain sustained exploration drive; when the set of reachable states is finite, the agent will eventually visit all reachable states.The experimental part evaluates the framework in sparse-reward mazes at three scales: 20×20, 30×30 and 40×40. The results show that DPFE reaches the goal in all seeds at 20×20 and 40×40, and in 4 out of 5 seeds at 30×30, and that the ratio of the number of steps per episode in the final stage of training to the shortest path length (computed only for seeds that successfully reach the goal) is 1.02×~1.17×, indicating that the agent has acquired a near-optimal path. Under the same interaction step budget, its state coverage is about twice that of the compared methods. In a continuous-state maze, after introducing a radial potential field the framework maintains a step efficiency of 1.09×~1.30× as the number of traversable states spans more than one order of magnitude.