Fine-tune the stable diffusion model using mindat data to generate mineral images from textual descriptions of attribute combinations

Abstract

Abstract Mindat (mindat.org) is a comprehensive mineral database that serves as a valuable resource for mineral data, encompassing a vast collection of textural descriptions and images from around the world. While the OpenMindat project accelerates knowledge discoveries in mineralogy, numerous relationships between images and text in Mindat remain underexplored. Exploring these connections could initiate a novel and intriguing discourse on transforming textual descriptions into mineralogical images, a field that possesses considerable potential value. This study aims to generate mineral images from textual descriptions of specific attribute combinations, with the requirement that the generated images conform structurally and visually to the specified attribute features. To implement the text-to-image generation, we first built a dataset of paired image-text data from Mindat, covering 6114 mineral species and 10 attribute types. We then fine-tuned the stable diffusion model utilizing two approaches: full fine-tuning and Low-Rank Adaptation (LoRA). We further evaluated both the baseline model and the fine-tuned models under these two strategies. The results show that the fully fine-tuned model surpasses both the LoRA and baseline models in terms of FID, KID, and CLIPScore, indicating a higher degree of similarity between the generated images and real-world mineral images. A practical application tool has also been developed, capable of interactively generating mineral images based on user-defined combinations of mineral attributes. Despite certain limitations, this tool provides an intuitive and controllable visual reference for mineralogists, enthusiasts, and science popularization educators, facilitating the inference of mineral visual characteristics from their intrinsic attributes.