Run-to-Run Consistency of Business Recommendations from Web-Search-Enabled Large Language Models: An Exploratory Study

Abstract

Consumers increasingly ask conversational large language models (LLMs) which product or company to choose, yet little is documented about how stable those recommendations are under repetition. We conducted an exploratory study of run-to-run consistency for two widely used web-search-enabled assistants, OpenAI's ChatGPT and Google's Gemini. Ten open "best X for a small business" questions spanning ten commercial categories were each issued three times to each assistant within a single day (60 responses total), with web search enabled throughout. From each response we extracted the set of recommended businesses and measured consistency as the mean pairwise Jaccard overlap of these sets across the three repetitions. Overall consistency was 69.5%, i.e. roughly one recommended business in three changed between identical queries. Consistency differed by assistant (ChatGPT 87.2% ± 17.5; Gemini 51.9% ± 11.8) but the instability was not uniform: of 127 distinct business-slots, 56% appeared in every repetition (a stable core) while 27% appeared in only one of three (a volatile tail). We discuss implications for reproducibility research and for the emerging practice of generative engine optimization, and we release the full method and response data. Given the small sample the results are directional, not definitive.