2026· International Conference on Software and Data Technologies· pp. 434-441· 0 citations· 8 references
Computer Science
TL;DR
It was observed that Portuguese prompts generally showed a slight advantage in terms of the accuracy rate compared to prompts in other languages, but the analysis of variance (ANOVA) indicates that this variation was not significant at the 5% level.
Abstract
: Artificial Intelligence (AI)-based code suggestion tools are becoming increasingly relevant in the software development community. While several studies analyse the quality and security of these tools, relatively few systematic studies have investigated how different input text languages can affect the effectiveness of code generation models. Similar research has been conducted with other tools, addressing multiple programming languages and different languages and challenge platforms. However, it is important to expand this knowledge base by considering additional scenarios. In this study, we analyse the performance of ChatGPT 5.2 in generating Python solutions for 70 programming problems extracted from the Beecrowd repository, considering three languages: Portuguese, English and Spanish. These 70 challenges were selected to encompass a range of difficulty levels for mathematical problems. Each question was submitted to the model five times in each language and the accuracy rate was evaluated using the platform’s validation tests. It was observed that Portuguese prompts generally showed a slight advantage in terms of the accuracy rate compared to prompts in other languages. However, the analysis of variance (ANOVA) indicates that this variation was not significant at the 5% level. The aim of this article is to extend current knowledge about the impact of languages on code generation by demonstrating that the language used in the problem description may or may not affect the ability of AI models to provide correct solutions. Finally, this article discusses the influence of potential linguistic biases, as well as opportunities for future research to address other languages, different problem modalities and various AI tools.
The synthesis of the reviewed studies indicates that AI-assisted feedback is frequently associated with improvements in grammar, vocabulary, coherence, and learner engagement, and several studies also highlight challenges, including concerns about academic integrity, students’ overreliance on AI, and the necessity for teacher guidance.
This work studies six functionally distinct web applications, each generated in two prompt variants that are identical except for an appended security-requirements section: a baseline (A) and a security-aware (B) variant.
GenAI can function as a research tool, but not as a substitute for methodological expertise, and has potential to increase efficiency of tasks which take advantage of its search and summarization abilities, as well as basic code debugging and algorithm formation.
Natalie Morosin, A. A. Nadi, Michael P Wallace· 0 citations
The results show that English prompts do not consistently produce the best functional correctness or code quality, the impact of prompt language depends on both the programming language and the LLM, and generated code frequently mixes English with the prompt language in comments and string literals.
Saima Afrin, Alessandro Midolo, C. Escobar-Velasquez et al.· arXiv.org· 0 citations
On the way of digitizing society, Artificial Intelligence (AI) has started to have profound impacts on all aspects of the modern life. In this regard, the main purpose of this study was to explore the effects of generative AI technology use on the students’ writing performance in language education context. Chat Generative Pre-trained Transformer (ChatGPT) chatbot developed by OpenAI was the AI-powered application utilized by 14 participants during their writing process. One-group pretest-posttest quasi-experimental research design was adopted as a study design. Based on the obtained data, it can be concluded that the use of ChatGPT chatbot with the purpose of developing writing skills have positive and significant effects on the aspects of grammatical accuracy, lexical sophistication, and syntactic complexity. It has been also noted that with its non-defensive learning opportunities, chatbot-assisted language learning has positive impact on the participants’ writing achievement, motivation, and self-directed learning.
Zeynep Yaprak· Journal of language research· 0 citations
Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.
Annemarie Wittig, Alina Mailach, Janet Siegmund et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
