Jul 2026· International Journal of Drug Delivery Technology· Vol 16· 0 citations· 11 references
TL;DR
An AI-based image generation system is introduced that utilizes a Stable Diffusion model fine-tuned with Low-Rank Adaptation (LoRA) for domain-specific image generation, demonstrating that SD+LoRA is an efficient and scalable domain-specific text-to-image generation system.
Abstract
Deep learning-based generative models have made a major leap forward in the world of image generation with the help
of Artificial Intelligence. One of the most notable of these developments is text-to-image synthesis, which can
automatically generate images based on natural language descriptions. In this work, an AI-based image generation system
is introduced that utilizes a Stable Diffusion model fine-tuned with Low-Rank Adaptation (LoRA) for domain-specific
image generation. The main idea of the proposed system is to combine the text encoding of CLIP, the latent compression
of Variational Autoencoder (VAE), and the denoising ability of diffusion to create images that are both semantically
relevant and visually coherent based on text prompts. The proposed approach was tested on a Pokemon image-caption
dataset for fine-tuning the pre-trained Stable Diffusion model and its effectiveness evaluated. The study shows that the
diffusion-based architectures outperform the traditional GAN based methods in terms of image quality, training stability,
semantic alignment, and output diversity. The main advantage of LoRA fine-tuning was the substantial decrease in
computational load, which involved updating just a small fraction of trainable parameters without compromising the
model's performance. Experimental results indicated that successful images of Pokemon could be generated, and that the
images were consistent with the text attributes such as color, type, and appearance. The results demonstrate that SD+LoRA
is an efficient and scalable domain-specific text-to-image generation system. The research underscores the rising
significance of diffusion-based generative AI in digital content creation, imaginative design, entertainment, and cleverness
in visual generation systems
Text-to-image generation is an increasingly fast-paced field of generative artificial intelligence, consisting of synthesizing images of high quality and semantic consistency based on natural language descriptions. In this paper, we give an extensive overview of the approach to text-to-image generation using deep learning, including the most common core model families, architecture designs, training approaches, and evaluation systems. We discuss the paradigms of the generative adversarial networks (GANs), variational autoencoders (VAEs), transformer-based designs, and diffusion models, with the last one representing the state of the art in image generation models. The review also discusses key aspects of pipelines such as text encoding, cross-modal alignment, mechanisms of attention, and decoding images. Popular datasets, methods, and metrics of evaluation, including Fréchet Inception Distance (FID) and CLIP-based similarity, are discussed. The application domains that involve creative content creation, medical imaging, education and industrial design are critically discussed. Despite significant advances, various issues still exist, such as low stability in training, excessive computational complexity, amplification of bias, generated images, and text–image alignment errors. Moral and social issues, such as misinformation, intellectual property, and equity, are critically examined. Lastly, we present future research directions to more controllable, more efficient and more interpretable text-to-image systems, focusing on multimodal foundation models and human–AI collaborative design.
Image caption generation is a significant area of study in artificial intelligence and computer vision, focusing on training systems to generate accurate textual descriptions of images. This paper presents an advanced framework for image captioning using a dataset sourced from Kaggle. The process begins with pre-processing techniques, including the Flexiframe Filter, which dynamically adjusts the window size based on local variance to reduce noise in both text and images. Bright- Contrast augmentation is then applied to enhance input images, enriching the dataset and improving feature extraction. The encoder module utilises a Sinusoidal Spatial Transform Convolutional Neural Network (SST-CNN) integrated with a Visual Geometry Group (VGG16) model for feature extraction and encoding. For the decoder, stochastic k-sampling is combined with a Dense Neural Network and a Gated Recurrent Unit (GRU) to generate initial captions. To refine these captions, a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model is employed for enhanced coherence and accuracy. The model's performance is evaluated using standard metrics, achieving BLEU-1, METEOR, ROUGE-L, and CIDER scores of 1, 0.99, 0.99, and 1, respectively. The proposed approach demonstrates significant improvements in generating descriptive and accurate image captions, making it a robust solution for applications requiring semantic understanding of visual data.
Bhargavi Polepalli, Praveen Kumar Sekharamantry, K. S. Rao· Pertanika journal of science...· 0 citations
This study investigates the automatic classification of real and AI-generated flower images using fine-tuned transfer learning models and shows that Swin Transformer-Tiny achieved the best overall performance, reaching an F1-score of 88.64% and outperforming the other architectures.
Unknown authors· NATURENGS MTU Journal of Eng...· 0 citations
The proposed Attention-Based Deep Learning Pipeline of AI-Created Image Recognition incorporates three integrated branches, including low-level statistical feature extraction, high-level semantic representation learning, and attention-based feature refinement mechanism, which support the robustness and generalization ability of the proposed model in detecting AI-generated images in a variety of generators and conditions.
Nadia Ali· Al-Noor Journal of Engineeri...· 0 citations
Text-to-image diffusion models have achieved remarkable success in generating high-quality images from a given text prompt. Subject-driven generation aims to synthesize customized images to mimic the appearance of subjects in given reference images within different visual contexts specified by the text prompts. The central challenge here is that, when the reference image changes, the diffusion model cannot efficiently adapt to different visual contexts while consistently maintaining the subject identity. Existing methods either train the model with a large domain-specific dataset or fine-tune the model using the reference image for hundreds of iterations before actual image generation. In this work, we explore a new approach, called \textit{In-Loop Model Adaptation} (IMA), which adapts the core diffusion model at each generation step during the actual process of image generation, without being trained on the reference image before the generation process. To this end, we establish a DDIM inversion chain that maps the reference image to a sequence of latent, as well as a text-to-image generation chain which generates the image from the text prompt only. We then introduce a masked latent consistency loss and a noise regularization loss to characterize the latent-noise difference between the diffusion model and these two chains at each generation step. This coupled latent-noise loss is used to guide the in-loop model adaptation to preserve the subject identity specified by the reference image while maintaining accurate alignment with the text prompt, resulting in high-fidelity text-to-image generation. Our extensive experiments demonstrate that our proposed IMA method significantly improves the performance of subject-driven text-to-image generation.
Yushun Tang, Weiming Chen, Siyi Liu et al.· IEEE transactions on multime...· 0 citations
Denoising diffusion models are the dominant architecture for image generation, whereas most natural language generation and modeling are primarily handled by well-known transformer architectures employing attention mechanism. Here, we show that diffusion models also inherently use an attention mechanism very similar to that of transformers. Therefore, attention emerges as a universal machine learning principle, based on a general training objective. We also show similarities in basic functional principle of auto-encoders and attention-based models. These equivalences allows us to interchange these designs based on practical requirements. As an example, we can reformulate the diffusion framework to reduce the lengthy training process and computation-intensive image generation. Using this approach, a simplified algorithm is proposed for image generation which is based on attention mechanism. Results show that the attention-based implementation achieves comparable performance with significantly less effort and computational resources.
F. Haddadi, L. Monfared, Ebrahim Rezaii et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
