The ChatGPT source cue shapes answer acceptance in a reasoning task

Abstract

As generative AI systems increasingly provide answers in everyday and academic contexts, understanding how source attribution influences evaluative judgment has become critical. In a within-subjects design, 103 participants evaluated 30 pre-generated reflective-correct, intuitive-incorrect, and arithmetic control-correct answers labeled ChatGPT or computer. Assignment of two parallel batteries to source labels and trial presentation order were randomized, and displays were visually matched. An independent unlabeled pretest with 35 participants supported battery comparability. Participant-level analysis showed greater overall acceptance under ChatGPT (68.0% versus 63.4%, p  = 0.002). A trial-level logistic mixed model with crossed participant and parallel-problem-pair random intercepts showed a source-by-category interaction ( p  < 0.001): acceptance was higher under ChatGPT for arithmetic controls (OR = 6.08, 95% CI [2.70, 13.69]), while source contrasts within the two CRT categories were not significant. Judgment accuracy is reported descriptively. Signal-detection analyses of CRT answers detected no significant source difference in sensitivity ( p  = 0.930) or criterion ( p  = 0.082). Evaluation times were longer under ChatGPT in the raw-time analysis ( p  = 0.034), but the source difference was not significant after log transformation ( p  = 0.180). Exploratory correlations detected no significant associations between source-related differences and trust, AI attitudes, or critical-thinking disposition. These findings distinguish overall source-related endorsement, particularly for arithmetic controls, from successful discrimination between correct and incorrect CRT answers.