Skip to content

The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations

Sep 2026 · 1 citation · 69 references
Computer Science

TL;DR

The Dice Roll Method is formalized as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling, giving repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation.

Abstract

Background: Researchers increasingly use repeated identical prompts to audit stochastic variation in large language model (LLM) brand recommendations, yet no standardized protocol exists for setting iteration counts, selecting stability metrics, or establishing reliability thresholds. Objective: We formalize the Dice Roll Method as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling. Methods: Total response variance is decomposed into sampling, prompt-phrasing, run-to-run, and model-version components. The stack: a negative-binomial mixed model with iterations as repeated measures; Cliff's delta as the distribution-free effect size; dependence-preserving bootstrap; simulation-based power; a generalizability-theory decomposition; drift diagnostics on pinned snapshots. We reanalyse five brand-recommendation auditing studies: approximately 190,000 observations, 270+ brands, 6 languages, iteration counts 5 to 40. Results: Three tiers of iteration guidance emerge from the D-study: exploratory (n = 5, G = 0.58), confirmatory (n = 10, G = 0.74), and rigorous (n = 15, G = 0.81), tied to effect-size and generalizability targets. The four metric families (count, set, embedding, fairness-adjusted PASOR) are complementary, motivating a compact metric battery over single indicators. A pre-registered external validation on three independent corpora (Motoki et al., 100-round; Rozado, 24 models; llm-stability) reproduces the D-study reliability prediction in 37 of 39 cells with no failures and the n = 5 power value to two decimals; the fixed tiers do not transfer, supporting a pilot-then-solve reading. Conclusion: The protocol gives repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation.

View source

Similar papers

Configuration, Not Conscience: A Large-Scale Empirical Study of LLM System Prompts

Analysis of leaked system prompts from 62 vendors across four community collections supports treating leaked prompts as operational specifications, closer to configuration files than value statements, and treats reuse and prompt rot as engineering and supply-chain concerns.

C. Patsakis, Vasileios Argyropoulos, E. Alepis · 0 citations
#natural language process... Preprint Sep 2026

Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit

Repeated-query audits must distinguish recovery of a collected set from completeness of possible outputs. We apply sample-based rarefaction to 4,500 responses from 50 buying questions, six configurations and 15 calls per cell. Historical-dictionary median ten-call recovery of the observed 15-call set ranges from 92.6%...

Dmitrij Żatuchin · 0 citations
#artificial intelligence Preprint Oct 2026

Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis

A repeat-run evaluation framework that aligns semantically equivalent categories and focuses on two operating metrics: theme churn, the normalized change in the returned category set, and volume disagreement, the change in counts for categories that persist shows that taxonomy grounding produces more consistent and rep...

Viraj Bagal, Raviraja Ganta, Prabhath Chellingi · 0 citations
Preprint Aug 2026

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy and pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases, a pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss.

Xiang-Fan Wu, Zong-Hao Ying, Hui-Yu Wu et al. · 1 citation
#natural language process... Conference Sep 2026

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spannin...

Mario Sanz-Guerrero, M. Bui, Manuel Mager et al. · 3 citations · ⚡1

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.