Experimental results on widely used SR benchmarks indicate promising improvements in perceptual quality and distributional realism with a compact trainable parameter scale, suggesting the value of reliable, stage-aware, and parameter-efficient multimodal prior integration for diffusion-based image super-resolution.
Abstract
Diffusion-based image super-resolution (SR) has recently achieved impressive perceptual quality by progressively generating plausible high-resolution details. However, its restoration performance still depends strongly on the reliability of the conditioning signal derived from degraded low-resolution inputs. Under severe or complex degradations, LR-derived conditions may become incomplete or ambiguous, leading the denoising trajectory toward visually plausible but input-inconsistent reconstructions. This work focuses on a central question: how to construct reliable multimodal prior guidance for a diffusion backbone that commonly adopts a hierarchical U-shaped architecture. To this end, we propose STMP-DiT, a stage-aware text-aligned multimodal prior-guided Diffusion Transformer for image super-resolution. From a multimodal data mining perspective, STMP-DiT aims to discover, align, and organize complementary semantic and structural priors from heterogeneous foundation-model representations. To improve the reliability of semantic guidance, STMP-DiT first aligns LLaVA-derived LR prompts with the frozen CLIP HR-image embedding space, producing visually grounded textual priors for restoration. These aligned textual priors are complemented by hierarchical DINO features, where deep features guide coarse semantic layout, intermediate features support structural recovery, and shallow features refine local edges and textures in the U-shaped DiT backbone. Rather than treating textual and visual priors as a single homogeneous condition, STMP-DiT assigns hierarchical DINO priors to different restoration stages according to their representational granularity. The fused condition is then injected through bounded feature modulation, enabling controlled stage-aware guidance while reducing redundant conditioning and improving the parameter efficiency of the conditioning modules. Experimental results on widely used SR benchmarks indicate promising improvements in perceptual quality and distributional realism with a compact trainable parameter scale, suggesting the value of reliable, stage-aware, and parameter-efficient multimodal prior integration for diffusion-based image super-resolution.
Existing diffusion-based enhancement methods provide strong generative capability for low-light image enhancement (LLIE), yet they either rely on paired supervision or lack reliable scene constraints in zero-shot settings, often leading to structural inconsistency and color drift. Motivated by conventional Retinex models, which offer physically interpretable priors that can serve as reliable scene constraints yet struggle with mixed degradations in real-world scenarios, we propose DARD, a zero-shot Degradation-Aware Retinex-guided Diffusion framework for LLIE. DARD first extracts image-specific physical priors from the degraded input through a test-time degradation-aware Retinex decomposition, thereby providing reliable structural guidance for zero-shot restoration. It then injects these priors into reverse diffusion through a timestep-adaptive frequency fusion strategy to balance structural anchoring and detail generation. Finally, a guided reverse refinement process with physical consistency and Contrastive Language-Image Pre-training (CLIP)-based semantic guidance is introduced to suppress structural artifacts and semantic drift during sampling. Extensive experiments show that DARD achieves strong distortion and perceptual performance and consistently outperforms existing zero-shot baselines across multiple real-world low-light benchmarks. To further validate the practical utility of our method for downstream applications, we evaluated its impact on semantic segmentation. Experiments demonstrate that images enhanced by DARD achieve a 28.10% relative improvement in mIoU over AGLLDiff.
Wenjie Cai, Yuezhe Yang, Jian-Yang Xia et al.· 0 citations
FSP-Diff, a novel one-step diffusion model featuring a dual-pathway architecture that refines semantic guidance using structured details to mitigate semantic deviations, is proposed and demonstrates that FSP-Diff surpasses existing one-step diffusion methods in both quantitative and qualitative metrics.
Chunxiao Liu, Wei Liu, Anbin Xiong et al.· 0 citations
Residual Flow Matching for Image Super-Resolution (RFMSR) is proposed, a vision-only framework that centers the source distribution at the LQ latent, reducing transport distance and preserving structural priors throughout the flow trajectory.
Shuwei Huang, Tianyao Luo, Jicheng Liu et al.· arXiv.org· 1 citation
Continuous diffusion models have become the dominant paradigm for photo-realistic image Super-Resolution (SR), but they typically formulate reconstruction as continuous signal-level denoising and incorporate semantic priors through external conditioning modules. This makes it less direct to exploit the unified token-based scaling paradigm of modern multimodal models. Autoregressive models provide a more native semantic representation by modeling images as discrete visual tokens, yet their causal decoding is inefficient for high-resolution reconstruction. Discrete diffusion offers a promising middle ground by enabling non-causal, parallel prediction over visual tokens. However, directly adapting discrete diffusion to SR remains non-trivial due to two task-specific challenges: (1) the long-tailed distribution of visual tokens, which under-represents rare but perceptually critical textures; and (2) spatially inconsistent parallel decoding, which may introduce isolated artifacts. To address these issues, we propose DiMOO-SR, a rarity-aware multimodal discrete diffusion framework for photo-realistic image SR. During training, Inverse Frequency Sampling (IFS) prioritizes under-represented but information-rich tokens. During inference, Spatial Consistency Ranking (SCR) refines token confidence using local neighborhood agreement to improve structural coherence. Extensive experiments on widely used real-world SR benchmarks demonstrate that DiMOO-SR achieves competitive perceptual quality with only a few parallel decoding steps, highlighting the potential of discrete diffusion for generative image super-resolution. The code will be released upon publication.
Ao Li, Yapeng Du, Yi Xin et al.· arXiv.org· 0 citations
A dual-path diffusion-based framework for IISR with a Global Representation Modulation module to extract modality-specific information from infrared imagery and guide the global distribution of the diffusion model toward the ground truth is proposed.
Yunpeng Hua, Hongwei Yu, Jiawei Li et al.· arXiv.org· 0 citations
UGDiff, a novel diffusion guidance paradigm designed to further improve the perception-distortion balance, is introduced, which first estimates the reconstruction uncertainty of the latent features corresponding to a high-fidelity image and guides the diffusion process to selectively restore high-frequency details in high-uncertainty regions, while preserving fidelity elsewhere.
Ren Wang, Yung-Yu Chuang· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
