Mapping the Thematic Network of Humanities Research Output Evaluation: A Mixed Method

Abstract

Purpose: Evaluating the outputs of the humanities has always been a fundamental challenge in scientific systems and research policy-making. Unlike empirical sciences, which often rely on quantitative and tangible criteria and indicators to assess their outputs, the humanities—due to their qualitative, interdisciplinary, and context-dependent nature—require a different evaluation model. Therefore, it is important and necessary to manage the evaluation of scientific outputs in the humanities in a manner that aligns with their unique characteristics. Although research has highlighted the distinct features of the humanities and the need for appropriate evaluation methods, as well as offered suggestions in this regard, the planning of such evaluations has received comparatively less attention. This study aims to develop a framework to enhance understanding of the evaluation process and its application in policymaking by systematically analyzing the interactions among the components involved in evaluating humanities outputs.Methodology: This research employs a mixed-methods approach, integrating both quantitative and qualitative techniques concurrently. Initially, a scoping review was conducted to identify and screen sources related to the evaluation of scientific outputs in the humanities. To locate potentially relevant documents, the Scopus and Web of Science databases were searched without any time restrictions. The selected sources were then analyzed using thematic analysis. To develop the themes, theoretical, inductive, and descriptive coding methods were applied to create a hierarchical structure comprising categories, domains, and taxonomies. Concurrently, we developed the conceptual structure of the data using bibliometric methods. The conceptual structure feature in Bibliometrix software (an R programming language package) is a tool for generating word co-occurrence networks (clustering), thematic maps (matrices), and thematic evolution over time (time course). As a quantitative method, it complements qualitative approaches. Finally, by integrating the results from these two stages, we created a network of themes related to the subject at hand. The thematic network, as an illustrative tool, summarizes the main themes of a text and reveals its underlying structure. It is presented graphically to make hierarchical concepts tangible, to provide fluidity among the themes, and to emphasize the interrelationships throughout the network.Findings: In the first stage, three taxonomies were identified: evaluation approaches, evaluation methods, and evaluation tools, each comprising different domains and categories. Evaluation approaches encompass five domains: evaluation fairness, conceptual consensus, impact citation, formative evaluation, and evaluation policy. Evaluation methods include three domains beyond the article level: evaluation criteria and peer review processes. Additionally, two domains—strong infrastructure and technological evaluation—are classified under evaluation tools. In the second stage, motor, niche, basic, and emerging themes, along with their changes from 2001 to 2025, were identified. Topics related to the evaluation of scientific outputs in the humanities were among the motor themes between 2001 and 2018; these themes were well-developed and highly interconnected. However, over time, these topics have tended to shift toward basic and niche categories. This shift suggests the embeddedness of these topics within research in this area and their broader recognition in the field of humanities evaluation. Between 2019 and 2025, the thematic clusters changed significantly, with the largest shift occurring from driving, fundamental-driving, and specialized-driving themes to other quadrants. In the third stage, bibliometric results were used to establish relationships between the themes identified in the first stage and the network of themes related to evaluating scientific outputs in the humanities. Across all three taxonomies—approaches, methods, and evaluation tools—there are internal relationships among the domains. Within the taxonomy of approaches, the domain of evaluation policy exhibits the most connections with other domains, highlighting its foundational role. In the taxonomy of methods, the domain of evaluation criteria is linked to the other two domains, reflecting the critical importance of how evaluation is conducted. Overall, these findings demonstrate that humanities evaluation is shaped by a complex network of interconnected domains rather than by isolated criteria, tools, or approaches.Conclusion: In policymaking, evaluation should function like a puzzle, meaning that approaches, methods, and evaluation tools must evolve simultaneously and proportionately. It is essential to analyze the strengths, weaknesses, opportunities, and threats of the current situation and to plan accordingly for the desired outcome. This shift in approach can lead to self-sufficiency in humanities evaluation at the policymaking level and reduce evaluation anxiety among researchers. These findings suggest that a more coordinated and context-sensitive evaluation framework can strengthen the legitimacy of humanities assessment and better align it with research policy goals.