GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.
As a paradigm in continual learning, class incremental learning (CIL) aims to assimilate tasks with mutually exclusive label spaces in sequence while preserving previously established knowledge. Mitigating forgetting in CIL fundamentally relies on transferring knowledge across tasks. A straightforward exemplar-based approach promotes balanced knowledge transfer by replaying an equal number of samples from each old class. However, in the more challenging exemplar-free setting, this balance cannot be ensured because distillation-based cross-task knowledge transfer tends to focus more heavily on the knowledge acquired from the most recent tasks. To address the unfairness in knowledge transfer, we analyze the mechanisms underlying dark knowledge and introduce a Semantic Enhanced Knowledge Transfer (SEKT) method for exemplar-free CIL. Specifically, SEKT adopts a bi-flow framework. The first flow is the Semantic Guidance Flow (SGF), which is inspired by knowledge distillation and produces latent semantic distributions from the outputs of the old model to guide the new model toward generating similar distributions. The second flow is the Semantic Propagation Flow (SPF), which propagates latent early knowledge to the current task in order to mitigate the unfairness in knowledge transfer. SPF constructs a cross-task semantic similarity graph using aligned intermediate representations to enable semantic propagation. It employs an expert network to learn the pattern of semantic propagation, enabling real-time and stable semantic recovery during training. In contrast to the SGF that is more effective for transferring recent knowledge, the SPF learns complementary early knowledge through a semantic complementarity constraint. Moreover, the SPF is robust to noisy semantics, as the learned semantic distribution is regularized with an $\ell _{2,1}$ norm. Extensive experiments conducted on six datasets demonstrate the superiority of the proposed SEKT over existing exemplar-free CIL approaches.
Fan-Kang Xu, Lu Jin, Yanpeng Sun et al.· IEEE Transactions on Image P...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
