Unified Multi-Weather Image Restoration with Intra-Task Difficulty and Inter-Task Contribution

Abstract

Recent studies have witnessed significant advances in unified multi-weather image restoration, which aims to handle diverse weather degradations within a single model. In this work, we observe that rain, haze, and snow restoration exhibit substantial differences in both degradation characteristics and learning dynamics, making straightforward joint optimization prone to performance imbalance and ineffective knowledge transfer. To this end, we propose UMWIR-Net, a unified multi-weather image restoration network equipped with an Asymmetric Task Collaborative Learning strategy. ATCL consists of Intra-Task Difficulty Optimization and Inter-Task Contribution Scheduling. Specifically, Intra-Task Difficulty Optimization jointly models the remaining restoration error and recent learning progress to dynamically estimate the optimization difficulty of each weather task, thereby assigning larger weights to slowly converging and under-optimized tasks. Inter-Task Contribution Scheduling measures the directional influence of a source-task update on the validation objective of a target task, constructs an asymmetric task-contribution matrix, and accordingly promotes tasks that provide stronger transferable knowledge while compensating those that benefit less from collaborative learning. In this manner, different weather restoration tasks collaborate selectively and asymmetrically, allowing the model to exploit complementary knowledge across tasks and improve overall restoration performance. Furthermore, UMWIR-Net adopts a wavelet-based Transformer backbone to capture low- and high-frequency information, enabling effective modeling of both global structures and local details for diverse weather restoration. Extensive experiments on multi-weather image restoration datasets show that UMWIR-Net achieves state-of-the-art performance and delivers more balanced restoration quality across rain, haze, and snow removal.