How Much Measurement Does an AI Visibility Claim Need? Replicate Instability and Minimum Detectable Effects in Generative Search Audits

Abstract

A commercial category now sells businesses a measurement of how often generative search engines cite them, typically from a few dozen prompts sampled once. We ask what such a measurement can actually support. Using 4,320 observations in which identical queries were issued three times to each of four generative answer surfaces (Google AI Overviews, Gemini with search grounding, Perplexity, and OpenAI web search), across six European languages and five commercial verticals, we quantify three things this literature does not report: the rate at which repeated identical queries disagree with themselves; how the width of a confidence interval on a citation rate falls with the number of prompts sampled; and the minimum difference two samples of a given size can distinguish from noise. Instability differs sharply by platform: from 1.4% to 40.3% of repeated prompt-platform cells return non-unanimous outcomes, meaning that on one widely used assistant a repeated identical question contradicts its own earlier answer two times in five. At twelve prompts, a plausible commercial configuration, the 95% interval on a platform citation rate spans 53 percentage points on the least precise platform, and the smallest resolvable difference between two samples ranges from 25 to 36 points. It follows that a monitoring product reporting month-over-month movement at that sample size is, for much of its range, reporting variation below its own detection threshold. We give a sample-size table practitioners and researchers can use directly, and release the observation log so the thresholds can be recomputed as the platforms change. We do not claim the underlying products are without value; we claim that the precision at which their numbers are presented is not supported by the sampling behind them, and that this is straightforwardly fixable by reporting intervals and replicate counts. Prompts were authored natively in each language and never translated. Tracked-brand domains are pseudonymised in this release; every reported quantity depends on the citation outcome rather than the identity of the brand.