A Seven-Criteria Framework to Evaluate Initiatives for Generative AI-Based Solutions

Abstract

Abstract The development and potential of large language models created incentives for organizations to adopt and integrate this technology in their internal processes. When the technological layer of organizations is in an acceleration phase there is a strong need to manage this change. Currently, generative artificial intelligence brings a vast area of opportunities, but this can create an “analysis paralysis” when deciding what initiatives to pursue. Different frameworks exist but they are targeting software solutions or specific domains. This paper follows an exploratory, framework-development qualitative design. An evaluation framework to assess initiatives based on generative artificial intelligence technology is developed by reusing existing decision models and enhancing them with criteria based on a simulated focus group using large language model personas. The interim findings were analysed using directed qualitative content analysis (deductive coding to a baseline codebook derived from literature), complemented by qualitative methods (free listing of evaluation criteria, anchored rating-scale elicitation, priority ranking, mapping to baseline criteria). A derived seven-criteria framework with definitions and performance scales resulted. This allows organizations to maximize chances of choosing the highest-reward initiatives and invest resources efficiently when needed to select, implement and integrate generative artificial intelligence solutions. The intention of this paper is to propose a list of seven evaluation dimensions — domain effectiveness and impact, regulatory compliance and ethical risk, user adoption, business value, technical feasibility, data availability and quality, and competitive differentiation that can be used to characterize applications based on this technology. [ms1][ms2]