Generative AI Accuracy and Its Effects on Reliance, Cognition, and Collaboration

Abstract

The proliferation of generative AI (Gen AI) tools (e.g., ChatGPT, Gemini, Claude) across society has led to advancements in various settings with reported benefits for individual productivity and collective problem-solving. The easy adoption of this technology has led to questions about over-reliance impacting individuals’ cognition and collective problem-solving, where individuals choose the least cognitively demanding path to a solution, without the necessary engagement with the content to determine its relevance or accuracy.A multi-phase mixed-methods study was designed to investigate the implications of AI accuracy levels on the propensity of individuals to rely on these tools, critical thinking about the content they produce, implications for creative processes such as convergent thinking, and collaborative problem solving. A bespoke app, which housed the surveys, cognitive tasks and the manipulated chatbot, was used for the study. The chatbots were manipulated to be predominately accurate, predominately inaccurate, or randomly varying. To assess carryover effects and potential shared mental models, a “mentor” chatbot was programmed for the group task.The results indicate that Gen AI accuracy influenced the level reliance on the tools as measured through prompting behaviours and propensity to use answers provided by the Gen AI tool. Further, the accuracy of the tool influenced the level of performance on various tasks, with detrimental effects for critical thinking and convergent thinking where the tool was not accurate. Finally, there was no evidence of priming effects across group conditions for problems solving or the use of AI.The study provides support for the growing understanding of Generative AI as a double-edged tool with clear performance gains when accurate, but potential damaging implications for cognition when inaccurate.