Skip to content

Continuous experimentation on artificial intelligence software: a research agenda

Nov 2020 · ESEC/SIGSOFT FSE · pp. 1513-1516 · 9 citations · 22 references
Computer Science

TL;DR

A holistic view of an iterative, continuous approach to develop industrial AI software basing on business goals, requirements and Minimum Viable Products is described and a research agenda with seven questions for future studies is proposed.

Abstract

Moving from experiments to industrial level AI software development requires a shift from understanding AI/ ML model attributes as a standalone experiment to know-how integrating and operating AI models in a large-scale software system. It is a growing demand for adopting state-of-the-art software engineering paradigms into AI development, so that the development efforts can be aligned with business strategies in a lean and fast-paced manner. We describe AI development as an “unknown unknown” problem where both business needs and AI models evolve over time. We describe a holistic view of an iterative, continuous approach to develop industrial AI software basing on business goals, requirements and Minimum Viable Products. From this, five areas of challenges are presented with the focus on experimentation. In the end, we propose a research agenda with seven questions for future studies.

View source

Similar papers

Preprint Aug 2026

Enactive Artificial Intelligence: A Decision-Centric Architecture for Complex Systems

By foregrounding decision intelligence in complex systems, Enactive AI expands the frontier of AI from model capability to system-aware action, opening new possibilities for scalable, governable, and socially valuable AI deployment.

Zuo-Jun Max Shen, Yuan Qu, Pu-Jun Zhang et al. · 0 citations
Open access Jul 2026

Supporting Traditional Manual Software Process With Aiware

The widespread use of artificial intelligence (AI) has been applied in many areas of application, including software engineering. In this practice-oriented study, AIware is adopted to build a traditional human-directed software model to eliminate manual software engineering development effort. The AI tools simplify and facilitate straightforward development processes, ranging from requirements specification, design, and testing. However, this paper incorporates human intervention to fine tune some important decisions during the development process because the AI tools may produce incomplete results. The effort spent on requirements creation by the proposed method is slightly reduced as the software size becomes larger. Future work is to enhance effective testing methods using innovative AI tools on a larger software development scale.

Nalinee Sophatsathit · 0 citations
2026

The AI Revolution

Artificial intelligence has evolved from specialized tools to comprehensive, goal-oriented systems that can learn, adapt, and soon even reason. New models that can write code, produce art, forecast market movements, and support decisions in all spheres of society are being developed every week. But this book covers more than just technology. Power exists. Ethics are a factor. It has to do with intelligence, the nature of future work, and people in general. It is up to the pioneers, leaders, and curious thinkers who must now lead us through the most disruptive time in our history. Dr. Jörg Storm is an international keynote speaker, podcast host, and lecturer at several academic institutions. He holds a Ph.D. in Economics and is recognized for turning technological complexity into crisp, board-level decisions and measurable business outcomes.

Jörg Storm · 0 citations
Book Open access Aug 2026

Enterprise AI Agents: From Prototypes to Production

Large language models (LLMs) have evolved from standalone generative systems into agentic AI systems capable of planning, reasoning, tool use, and multi-agent collaboration. Enterprises are increasingly adopting AI agents to automate and orchestrate complex workflows, from IT operations to employee productivity. While early deployments focused on proof-of-concept prototypes, the past year has marked a clear shift toward production-grade enterprise AI agents. This transition has been enabled by a wave of new technologies, including multi-agent orchestration, memory and state management, skill-based and modular agent architectures, and deeper integration with enterprise data and workflow platforms, which together make scalable, reliable agent systems feasible in practice. At the same time, moving agents into production introduces new technical and organizational challenges, such as rigorous evaluation and benchmarking, security and governance, and system design for long-running, autonomous operation. Building on the success of our two prior highly attended editions: ''Agentic AI for Enterprise'' workshop at KDD 2025 and ''Enterprise RAG'' workshop at CIKM 2024, this workshop aims to bring together researchers and practitioners to examine how enterprise AI agents can successfully move from prototypes to production. We focus on three pillars: 1) Agent architectures and systems; 2) Enterprise applications and deployments; 3) Evaluation and governance.

Min Du, Anbang Xu, Jasmine Jaksic et al. · 0 citations

Towards Autonomous Software Development

A three-level taxonomy inspired by autonomous driving that distinguishes degrees of autonomy along a roadmap from today’s AI-assisted development workflows to fully autonomous software development in which AI systems autonomously identify demands and design, implement, verify, and maintain software without human oversight is introduced.

Hao Wang, Ruijie Meng, Zhe Ye et al. · 0 citations
Review Open access Aug 2026

Could it be more than a toolbox? Defining a business model for artificial intelligence-driven general contractors

The purpose of this research is to address the potential of artificial intelligence (AI) to drive business models for general contractors. While AI is used for various tasks, its holistic impact on general contractors' business models remains unclear. The research utilizes a qualitative, multi-stage, action-research-oriented study. The methodology includes a literature review, two internal company workshops to design a specific business model and a third validation workshop with 25 industry experts to broaden and generalize the model. The research provides a transferable, AI-driven business model structured around Osterwalder's business model canvas components. It details a gradual adoption pathway for general contractors, addressing the tension between AI as an operational tool and as a transformative force. The pathway starts with internal efficiency enhancements (an incremental “toolbox” approach), progressing to value-added client services and culminating in transformative, servitized offerings such as project-independent AI-as-a-Service (AIaaS). This study addresses a significant gap – AI-driven business models for general contractors are largely missing from the existing research. It moves beyond viewing AI as a toolbox for isolated tasks to explore its potential to reconfigure the entire business system. The research provides a timely and structured overview that is valuable to both industry and academic actors by conceptualizing the AI-driven contractor as a data-driven ecosystem orchestrator.

R. Nyqvist, Mikko Kuusakoski, Sakari Aaltonen et al. · 0 citations

Related blog posts

Microsoft Research Blog Jul 30, 2026

Echoverse: Deep, evolving environments for computer-use agents

Computer-use AI agents struggle with multi-step workflows like email and customer support. Echoverse trains agents in realistic environments rather than simply providing more training tasks, helping them improve as the tasks, tests, and environments evolve. The post Echoverse: Deep, evolving environments for computer-use agents appeared first on Microsoft Research.

MIT News · Artificial Intelligence Jul 14, 2026

Helping AI models to meet the real world

Through research and entrepreneurship, Professor Devavrat Shah is helping to design methods that can handle constant decision-making using limited computational resources.

MIT News · Artificial Intelligence Jun 3, 2026

MIT researchers teach AI models to interpret charts

The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.