Experiments show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions.
Abstract
Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/
Recent advances in text-to-image (T2I) generation have enabled controllable image synthesis by incorporating conditions beyond text. However, most existing diffusion-based methods are limited to a single type of control condition (e.g., bounding boxes or keypoints), which restricts their flexibility. To address this limitation, we propose MixDiffusion, a training-free diffusion framework for multi-condition T2I generation. MixDiffusion theoretically supports an arbitrary number of control conditions, including bounding boxes, keypoints, sketches, depth maps, reference images, and text, by collaboratively integrating multiple pre-trained uni-condition diffusion models. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-condition image generation model from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. Owing to its training-free nature, MixDiffusion is easy to deploy and readily extensible to new control modalities.
Pengcheng Wan, Liang Han, Lin Xu et al.· arXiv.org· 0 citations
In the task of text-to-image synthesis, it is challenging to generate semantically consistent and high-fidelity images from given text descriptions. To address the issues of semantic inconsistency between text and images and the lack of realism in generated image details, this paper proposes a GAN-based method for generating images from text. Specifically, by introducing multi-kernel convolution into the generator and replacing standard convolutions with multi-layer nested convolutions combined with dynamic gating weighting, the model is encouraged to focus on crucial detailed features, thereby improving the fine-grained quality of generated images. Then, we design a feature enhancement block that models multi-dimensional feature interactions via channel, spatial height, and spatial width branches, which effectively enhances feature representation and produces more photo-realistic images. Furthermore, we propose a fine-grained interaction mechanism that utilizes a low-rank correlation matrix to capture the dependencies between global and local information at different granularity levels, achieving fine-grained channel enhancement and significantly improving text-image alignment accuracy. Moreover, the proposed model maintains the advantages of efficient generation and a smooth latent space from the GAN paradigm. Finally, comprehensive experiments are carried out on the CUB and COCO benchmarks. Quantitative results illustrate that our approach achieves 11.48 FID and 5.23 IS on the CUB dataset, and 17.46 FID and 36.03 IS on the COCO dataset, respectively.
Pengkui Wang, Xiao-Sheng Cai· International Conference on...· 0 citations
Existing texture generation methods rely heavily on latent diffusion models, whose VAE-based spatial compression inherently limits fine-grained detail preservation and degrades pixel-level multi-view consistency. To address this limitation, we introduce PixTex, the first pixel-space multi-view diffusion framework for texture generation, which achieves substantially improved multi-view consistency. Operating directly in image space avoids latent compression, reduces inconsistencies introduced during latent-to-RGB upsampling, and preserves lossless pixel-level geometric guidance for accurate multi-view consistency. However, directly applying pixel-wise attention across multiple views is computationally prohibitive. To balance efficiency and fidelity, we adopt a coarse-to-fine consistency strategy: i) At a coarse patch level, we establish cross-view structural correspondence by employing 5D RoPE to correlate 2D patch coordinates with 3D world-space positions. ii) At the pixel level, a specialized 3D position-aware detailer further refines textural details based on patch features, ensuring fine-grained alignment unattainable by VAE-based methods. Additionally, we propose a novel consistency loss to explicitly guarantee multi-view coherence. Finally, we incorporate a pixel-space multi-view inpainting module to resolve self-occlusions and improve texture completeness. Extensive experiments demonstrate that our framework achieves state-of-the-art multi-view consistency, producing high-fidelity and seamless textures.
Yuqing Zhang, Yan-Pei Cao, Hao Xu et al.· International Conference on...· 0 citations
Pixel-space diffusion has recently emerged as a promising direction for high-fidelity image generation by modeling images directly in the original pixel domain. However, pixel-space diffusion is computationally demanding due to the extremely high dimensionality of natural images. For efficiency, existing pixel diffusion models either compromise fine details with large-patch tokenization or confine subsequent refinement within individual patches. Such intra-patch refinement inevitably restricts structural continuity across patch boundaries and long-range token interactions, limiting refinement quality. To address these issues, we propose PixSGR, a novel Pixel diffusion framework with Sparse Global Refinement tailored for modeling the distribution of natural images directly in pixel space. PixSGR starts from a supervised low-channel bottleneck to efficiently capture the low-dimensional manifold of natural images. It then progressively expands the channel dimensionality and spatial resolution to recover increasingly fine-grained structures. At the spatial refinement stage, coarse-scale attention maps preselect globally relevant interactions to pre-sparsify fine-scale attention, enabling non-local refinement beyond isolated patches without the quadratic cost of dense attention. Extensive experiments on ImageNet validate the effectiveness of PixSGR. It achieves an FID of 1.51 at 256$\times$256 and maintains performance when scaled to 512$\times$512, attaining an FID of 1.60.
Weiyi You, Jin-Hua Zhang, Xing-Yu Zhou et al.· 0 citations
Text image super-resolution aims to improve the readability of low-quality text images while preserving character structures, stroke details, and semantic consistency. Compared with natural image super-resolution, this task is more sensitive to structural distortion because small changes in stroke topology may lead to incorrect text recognition. To address this problem, this paper proposes an OCR prior-guided cross-scale framework for text image super-resolution. Specifically, character-level semantic priors extracted from a pretrained OCR model are introduced to provide structural guidance for degraded text reconstruction. A gated feature modulation mechanism is designed to adaptively regulate the contribution of OCR priors, reducing the influence of unreliable semantic predictions. A cross-scale dynamic attention module is also developed to aggregate multi-granularity visual features, enabling the model to jointly recover fine stroke boundaries and global character structures. In addition, a sequence-aware calibration module is introduced to improve structural consistency along the logical reading order of text. Experiments on mixed text image benchmarks and the TextZoom dataset show that the proposed method achieves competitive or better performance among the compared methods in terms of PSNR, SSIM, and recognition-oriented metrics. Additional ablation, OCR prior robustness, and computational complexity analyses further indicate that the proposed framework improves text readability while maintaining a reasonable accuracy–complexity trade-off. The results also suggest that OCR priors are useful for text image reconstruction, but should be used as soft constraints when external recognition predictions are uncertain.
PoseAdapter, a lightweight framework for high-fidelity 2.5D controllable image generation, and a Context-Aware Dual-Stream Representation, to resolve the generative trade-off between strict instance isolation and global coherence.
Yufeng Chi, Hui-Min Ma, Fan Gao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
