GenAI can function as a research tool, but not as a substitute for methodological expertise, and has potential to increase efficiency of tasks which take advantage of its search and summarization abilities, as well as basic code debugging and algorithm formation.
Abstract
Generative artificial intelligence (GenAI) is a large language model (LLM) that has the ability to generate media based on user-provided prompts. Given the demonstrated capabilities of models such as ChatGPT in information synthesis and programming, there is growing interest in their potential role within the research process. However, little work has evaluated recent GenAI models for research tasks in the domain of statistical research. This case study examines GenAI as a tool for developing a literature review and translating methodology from academic papers into code, for the topic of dynamic treatment regime (DTR) estimation via the dynamic weighted ordinary least squares (dWOLS) approach. Specifically, we utilize ChatGPT-5 and ScholarAI (Sept-Nov 2025 release) in the processes of identifying relevant sources for the literature review, creating summaries of papers, identifying gaps in research, and R code generation to implement methodology. Our findings show that current GenAI models lack the depth and contextual understanding required to accomplish these tasks without careful prompting and supervision of a knowledgeable researcher. Nonetheless, GenAI has potential to increase efficiency of tasks which take advantage of its search and summarization abilities, as well as basic code debugging and algorithm formation. We demonstrate that under a knowledgeable guide, GenAI can function as a research tool, but not as a substitute for methodological expertise.
BACKGROUND
Large language models have accelerated the adoption of generative artificial intelligence (AI), making AI tools more widely accessible through conversational prompting. One emerging application is vibe coding, in which users use natural-language prompts to generate code and desired outputs rather than manually writing traditional code.
OBJECTIVE
To examine AI-assisted, human-in-the-loop (HITL) vibe coding as a proof of concept for data analysis, describe its components and a proposed workflow with explicit safeguards, and present a case study illustrating its use and potential failure points.
METHODS
We used a proposed workflow that included framing research questions, operationalizing variables, organizing project folders, documenting decisions, applying retrieval-augmented generation, and using prompt engineering techniques. We used Cursor (v1.5.11) on a limited, clean admissions data set, in which admission status was modeled as a function of the Graduate Record Exam, grade point average, and undergraduate rank. Logistic regression was generated via conversational prompts, implemented in R, and the results were compared with a published reference output on a publicly available website.
RESULTS
AI-assisted, HITL vibe coding produced statistical codes that included schema checks, range validations, data cleaning, exploratory analyses, regression modeling, and visualization. There were mixed results of both valid and invalid outputs. Regression coefficients, p values, and model fit statistics matched the outputs posted on the published reference output website. However, an error was identified in the predicted-probability confidence interval output, which was missed during the initial review of outputs.
DISCUSSION
While vibe coding has the potential to reduce barriers to data analysis for researchers, this case study demonstrated that it can produce both valid and invalid outputs and that foundational statistical training, knowledge, understanding, and methodological expertise remain paramount when using it. Future studies should address important empirical questions about the use of vibe coding, such as under what conditions it can be safely used in research and what kinds of errors are most commonly generated when using it. AI-assisted HITL vibe coding should be used with caution and only with structured verification and safeguards, transparent reporting, and appropriate statistical and methodological oversight.
D. Tolentino, E. Kohout, Paul Boy et al.· Nursing Research· 0 citations
Whether contemporary LLMs can reproduce the research outcomes of a fully documented human study: a 1991 article that identified dermatophytosis (ringworm) in historical fine art was evaluated.
Recent advances in Generative Artificial Intelligence (GenAI) have substantially improved the ability of large language models (LLMs) to generate and explain source code. However, their performance on authentic object-oriented programming (OOP) assessments remains insufficiently understood. This study evaluates five widely used GenAI systems, ChatGPT-5.2, DeepSeek-V3, Gemini 2.5 Flash, Claude Sonnet 4.5, and M365 Copilot, using programming tests and examination tasks from an introductory university OOP course. The generated solutions were assessed using the same grading criteria applied to students and compared with historical student results from the same course, as well as findings from the previous year. Common errors were also analyzed to identify recurring limitations across models. All evaluated GenAI systems achieved higher scores than the average student cohort and frequently obtained full marks on longer programming tasks. Nevertheless, they occasionally produced non-compiling code and continued to struggle with advanced OOP concepts, particularly interfaces, abstract classes, and certain inheritance-related tasks. Performance was also limited on graphics-related questions involving image interpretation. Compared with the previous year, the evaluated systems demonstrated noticeable improvements across most assessments while exhibiting several recurring error patterns. The findings provide an updated evaluation of the capabilities and limitations of contemporary GenAI systems on authentic introductory OOP assessments. They also offer evidence that can inform the design of programming assessments, the responsible integration of GenAI tools into software engineering education, and future studies evaluating the evolution of AI-assisted programming.
It is concluded that AI meaningfully augments developer productivity but does not yet demonstrably improve satisfaction or earnings, and that a hybrid human-AI model, supported by governance and training, remains the most defensible direction for application development.
Perseus Bhavnagri· International Journal for Re...· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
It is established that LLM outputs should be treated as draws from a distribution rather than as fixed measurements, as well as guidance for data editors and authors.
Guillaume Coqueret, J. Llull, Florian Oswald et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
