PMRF-VTON: diffusion-based virtual try-on with pose–mask region modulation and frequency-domain residual guidance

Abstract

Virtual try-on aims to naturally dress a target garment onto a given human image, and has broad application value in e-commerce, digital humans, and intelligent garment design. Methods based on latent diffusion models have substantially improved the realism and generation stability of try-on images, yet two limitations remain. First, the denoising process insufficiently models the spatial relationships among human pose, garment region, and inpainting mask, which easily causes boundary misalignment and unnatural blending at the neckline, cuffs, and occluded regions. Second, latent-space compression and step-by-step denoising weaken the high-frequency information of garments, making fine details such as text, logos, and stripes difficult to preserve accurately. To address these issues, this paper proposes PMRF-VTON, a diffusion-based virtual try-on method that integrates pose–mask region modulation with frequency-domain residual guidance. Two lightweight enhancement modules are introduced into the denoising UNet of Stable Diffusion: the Pose- Mask Adaptive Region Modulation (PMARM) module jointly exploits human pose, the inpainting mask, and garment latents to apply region-adaptive affine modulation to intermediate features, thereby strengthening the structural consistency of garment boundaries and occluded regions; the Residual Fourier Guided Module (RFGM) explicitly models the amplitude and phase of hidden features via the Fourier transform and enhances high-frequency textures and edge details in the form of a residual adapter. Both modules are injected into the frozen pre-trained backbone as plug-and-play residual branches, jointly enhancing the generation process from the two complementary perspectives of structure and texture. Experiments on the VITON-HD dataset show that PMRF-VTON achieves competitive results in terms of SSIM, LPIPS, and FID.