Can multimodal large language models evaluate AI-generated industrial design images? An empirical comparison with human expert assessment

Abstract

Text-to-image (T2I) generation models are increasingly used in industrial design, but assessing the quality of their outputs still depends on slow, small-scale human expert review, which constrains the automation advantage of generative AI and hinders its adoption in this domain. This paper empirically examines whether multimodal large language models (MLLMs) can substitute for design experts in evaluating AI-generated product design images. We construct a dataset of 120 images generated by three T2I models (Midjourney v6, Stable Diffusion XL, GPT-Image-1) across four product categories, and define a four-dimensional rubric — aesthetic quality, prompt fidelity, creativity, and design feasibility — anchored in design-assessment and T2I-evaluation traditions. Four MLLM judges and seven trained design experts rated all images under the identical rubric in pointwise and pairwise modes. In overall agreement, the two top-performing AI evaluators aligned with the expert consensus more closely than individual human evaluators did, with Spearman ρ of 0.51 and 0.42 compared to a human baseline of 0.36 obtained by the leave-one-out method. Furthermore, their pairwise judgments agreed with the majority opinion of the experts 65–69% of the time (p ∠ 0.001). However, this consistency was not uniform across evaluation dimensions: agreement was lower in the aesthetic-quality dimension, where human evaluators themselves also showed reduced consistency. Based on these findings, we suggest practical guidelines for human–AI collaborative evaluation workflows in design practice.