A Text-Guided Cross-Modal Diffusion Framework With Attention-Based Hand-Drawn Sketches for Face Synthesis

Abstract

Face sketch-to-photo synthesis plays a key role in computer vision with applications in law enforcement, digital entertainment, and human–computer interaction. Existing generative adversarial network-based methods typically face mode collapse, training instability, and poor performance across different sketching styles. This study introduces Sketch-to-Face, a cross-modal diffusion-based model that leverages Stable Diffusion to generate photorealistic faces using sketch-based inpainting. The proposed approach comprises three components: a Sketch Encoder with multiresolution attention that produces CLIP-compatible embeddings from grayscale sketches, Cross-Modal Fusion module employing bidirectional attention to bridge sketch spatial features with text semantic features, and Automatic Mask Generator with learnable refinement for adaptive inpainting guidance. A low-rank adaptation was applied to fine-tune the U-Net, reducing trainable parameters to 20.5 M while retaining the 860M-parameter backbone. Experiments on the Person Face Sketches dataset (21K+ pairs) show stable convergence and are evaluated using SSIM, LPIPS, FID, identity similarity, runtime, and GPU memory. The results demonstrate a favorable quality–efficiency trade-off compared with text-and-sketch diffusion baselines, while qualitative results indicate plausible facial structure and visual realism.