TL;DR
A reverse engineering technique is presented for extracting a behavioural model of a deployed chatbot through the automated exploration of its functionality via Large Language Models through the automated exploration of its functionality via Large Language Models.
Task-based chatbots are nowadays widely adopted software systems, usually integrated into real-world applications and communication channels, designed to assist users in completing tasks through conversational interfaces. Like any other software, even chatbots are prone to bugs. Despite their increasing pervasiveness in everyday activities, existing techniques for assessing their quality still exhibit several limitations, such as the simplicity of generated test scenarios and oracle weaknesses. In this paper, we present LexTester, an automated model-based testing technique for Amazon Lex chatbots. The technique explores the conversational space of the chatbot under test to generate a Dialog Graph of all possible interactions, from which an executable test suite is generated according to different coverage strategies. LexTester was evaluated against the state-of-the-practice chatbot testing tool Botium on five Amazon Lex chatbots, consistently outperforming it in all subjects, generating more tests with nearly double complexity, achieving overall 83-95% coverage of conversational elements, and improving fault detection effectiveness by up to four times at comparable time costs.
The rapid evolution of Large Language Models (LLMs) has opened new possibilities for building intelligent, human-like conversational systems capable of understanding and responding to open-domain queries in real time. Traditional chatbot architectures, whether rule-based or narrow-domain trained, are constrained by limited contextual understanding, poor conversational continuity, and an inability to generate fluent, human-like responses across varied topics. This paper presents the design, development, and evaluation of an AI-Powered Conversational Web Assistant that integrates Google's Gemini 1.5 Flash large language model with a lightweight Python Flask web application to deliver real-time, context-aware conversational support. The system accepts free-form natural-language input through a responsive browser-based chat interface, forwards the query to the Gemini API through secure RESTful calls, and renders the generated response asynchronously without page reload. A Flask backend manages session handling, prompt construction, and API communication, while a SQLite-backed logging layer stores conversation history for auditing and future personalization. The architecture is deliberately modular, separating the presentation layer, the dialogue-management logic, and the external LLM service, which allows any of these layers to be replaced or extended independently. Experimental use of the deployed prototype across a range of factual, technical, and conversational queries demonstrated that the assistant produces fluent, contextually relevant, and largely accurate responses while maintaining low perceived latency. The study also documents the practical challenges encountered during development, including prompt-engineering trade-offs, API rate limits, session-memory handling, and safe-response filtering, together with the mitigation strategies adopted. The resulting prototype confirms that a cloud-hosted multimodal LLM, when combined with a minimal and well-structured web stack, can serve as a practical foundation for next-generation digital assistants suitable for customer support, education, and personal productivity applications, with a clear pathway toward voice interaction, sentiment-aware responses, multilingual support, and persistent contextual memory in future iterations. Index Terms: - Artificial Intelligence • Natural Language Processing • Large Language Models • Google Gemini API • Conversational AI • Chatbot • Flask Framework • Prompt Engineering • Intent Recognition • Dialogue Management • Natural Language Generation • SQLite • Human-Computer Interaction • Web-Based Assistant.
Academic and student services at Eastern Samar State University–Salcedo Campus traditionally rely on manual processes. These methods frequently lead to response delays and restrict information availability outside of regular operating hours, hindering students' timely access to crucial campus data. This study aimed to design, develop, and evaluate an AI-driven, intelligent chatbot tailored to support student service automation at ESSU-SC, focusing on routine, front-facing queries such as enrollment steps, schedules, and general university policies. Employing a developmental research design, the system utilized Google Dialogflow for Natural Language Processing and integrated AWS services alongside OpenAI APIs to build a dynamic conversation workflow. System interfaces were deployed via official university web platforms and Facebook Messenger. A rigorous evaluation framework was executed across three sequential phases: a Benchmark Test, an Alpha Test, and a Beta Test using the ISO 9126 Quality Model, culminating in a System Usability Scale (SUS) evaluation. Evaluators included system design experts from the faculty, standard university students, and a visually impaired individual to verify accessibility features. The system demonstrated progressive quality enhancements across all metrics throughout the testing lifecycle. The initial Benchmark Test yielded a "Very Good" overall mean score of 3.68, which advanced to an "Excellent" rating of 4.76 during the Alpha Test phase following interface and architectural refinements. The real-world Beta Test achieved a mean score of 4.38 ("Very Good"), showcasing perfect efficiency scores (5.00) in real-time response times and data processing. Furthermore, the SUS yielded a final score of 78, confirming that the application is highly usable, practical, and effective as an inclusive, assistive communication tool. The study successfully validated that the intelligent chatbot efficiently bridges communication barriers, automates repetitive administrative inquiries, and improves service accessibility. Future recommendations include expanding the bot's scope to encompass all university services, introducing multi-language capabilities, and adding advanced accessibility features to support broader inclusion.
Voice assistants are widely deployed on mobile platforms, yet most are designed as system-level services that remain poorly aligned with application-specific behavior. As a result, enabling voice interaction at the app level requires developers to manually reimplement application logic, leading to high development and maintenance costs. We propose an LLM-driven approach to automating the development of app-specific voice assistants by repurposing GUI test code, which encodes behavior-preserving, executable specifications of application functionality. In this paper, we present a perspective in which large language models reinterpret GUI tests as bridges between application behavior and conversational interaction. By transforming test methods into app-specific VA artifacts, such as voice intents, capability descriptions, and executable action plans, our approach grounds voice assistants directly in existing application logic rather than external specifications. We illustrate this vision through AppVA, a research prototype on Android. Our preliminary results across five open-source applications suggest that GUI test code can be reused beyond testing, enabling the synthesis of app-specific voice assistants and highlighting a broader research direction at the intersection of software testing, interaction design, and LLM-enabled automation.
Agents built on large language models (LLMs) are increasingly used to build applications that perform complex, multi-step tasks involving reasoning, tool use, and interaction with external environments. Despite rapid progress in benchmarking LLM-based agents, very few studies have attempted to understand how such systems are tested in practice. In particular, testing levels, objectives, data patterns, test complexity, and validation strategies for agent applications remain underexplored. In this paper, we present an empirical study of testing practices in LLM-based agent applications using a large corpus of mined open-source projects. We construct a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules. From this analysis, we derive a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level (unit, module, integration). We complement this with structured interviews of 10 senior industry practitioners building agentic systems. Our results show that testing of LLM-based agent applications is dominated by narrowly scoped unit tests, with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. Tests frequently rely on simplistic inputs, heavy mocking, and shallow validation, and agent-related tests exhibit low structural complexity. While industry practice places greater emphasis on non-functional testing than open-source projects, both reveal common gaps, including the lack of formal testing foundations, unclear test objectives, and challenges in generating high-quality test data. Based on these findings, we outline research directions toward more systematic and rigorous testing of agent applications, including foundations for agent testability, formalized test objectives, and fault-based testing techniques.
Conversational AI assistants such as Siri, Google Assistant, and ChatGPT have improved how users interact with digital systems, yet they remain confined to answering questions rather than completing real-world tasks. Users must still manually open applications, navigate websites, fill out forms, and switch between software to finish even simple workflows, which limits productivity and creates accessibility barriers for elderly users, first-time computer users, and individuals with disabilities. This paper presents TEXA OS (Trusted Executive Assistant), a self-improving agentic artificial intelligence platform designed to function as a complete digital executive assistant rather than a conversational chatbot. Unlike static automation scripts built with tools such as Selenium or Playwright alone, TEXA OS combines natural-language intent understanding, autonomous task planning, and multi-agent execution to decompose a single spoken or typed request into an ordered set of executable subtasks. A FastAPI-based AI orchestrator interprets user intent using a large language model and delegates the resulting subtasks to specialized execution agents responsible for browser automation, operating system control, document generation, and communication automation. The system is built on a React and TypeScript frontend with continuous voice-recognition support through the Web Speech API and a PostgreSQL backend that persists user preferences, task history, and long-term AI memory, allowing the platform to adapt future task execution based on prior interactions. A distinguishing capability of TEXA OS is its Website AI Navigation Module, which allows users to retrieve information from complex websites—government portals, banking systems, and educational. The system was evaluated across unit, integration, and system-level test cases spanning voice recognition, browser automation, document generation, and permission-gated execution, achieving an overall task-execution success rate of approximately 96% and an overall system reliability of approximately 86%. The results indicate that agentic orchestration combined with permission-based safety controls provides a practical foundation for autonomous, trustworthy task automation across operating systems and web environments. Keywords: Agentic AI, Task Automation, Large Language Models, Browser Automation, Voice Recognition, AI Orchestrator, Natural Language Processing, FastAPI, Self-Improving Systems, Human-Computer Interaction.
In his latest book, Professor Eugene Fitzgerald examines the forces that turn breakthroughs into value — and why innovation resists simple formulas.
MIT students designed, built, and tested a jet engine with AI copilots, assessing AI’s usefulness in developing high-performance aerospace systems.
Researchers developed an auditing technique to test generative AI models for malicious capabilities, without prompting them for illegal outputs.
A USAF cadet and a Lincoln Laboratory researcher found AI chatbots can help nontechnical service members produce viable software applications for their unique problems.
