Investigating the Potential of Large Language Models to Support Qualitative Data Analysis - Replication Package

Abstract

Replication Package: Investigating the Potential of Large Language Models to Support Qualitative Data Analysis This repository gathers the complete replication package and open data regarding the master's dissertation entitled: Investigating the Potential of Large Language Models to Support Qualitative Data Analysis. The research analyzes the behavioral patterns, inferential limitations, and Prompt Engineering strategies of Large Language Models (LLMs) — specifically Google Gemini 2.5 Pro — in supporting deductive qualitative coding under the continuous supervision approach (Human-in-the-Loop). The investigation is based on two experimental cycles in sociotechnical domains of Software Engineering: Cycle 1 (Base Study - Empathy): 238 discursive excerpts across 4 analytical dimensions (Meaning, Barriers, Practices, and Effects) under a multi-label classification regime with 54 codes. Cycle 2 (Replication Study - Psychological Safety): 100 excerpts in 2 dimensions (Interpersonal Challenges and Practices) under a single-label classification regime with 18 codes. Package Content and Directory Structure The artifacts are structured to ensure transparency, auditability, and immediate reproducibility: 01_corpora: Textual excerpts extracted from professional communities (DEV Community and SE-SWE) in raw and pre-processed formats, indexed by unique identifiers. 02_codebooks: Complete reference codebooks for Empathy and Psychological Safety, containing operational definitions and explicit inclusion/exclusion rules. 03_prompts: Standardized instruction templates structured into 6 components (Persona Definition, Contextualization, Conceptual Codebook, Explicit Instructions, Structured Output with Mandatory Justification, and One-Shot and Few-Shot Anchoring Examples). 04_scripts: Python automation code (via official Google Generative AI library), including rate control (sleep time), segmentation in batches of 10 excerpts, and CSV persistence routines. 05_raw_model_outputs: Raw and immutable records of the inferences generated by Gemini 2.5 Pro, with assigned codes and their respective complete textual justifications. 06_qualitative_analysis_and_irr: Initial inter-rater reliability (IRR) data, independent blind evaluation matrices, consensus spreadsheets, and complete categorization according to the Taxonomy of Concordance and Divergence (5 classes). 07_expert_panel: Focus Group moderation protocol, slide decks for blind case presentations, full transcripts, and deliberation reports from senior experts. Main Contributions Provided Framework for Assisted Deductive Coding Support (SCDA): Methodological guidelines structured on three pillars (Input Rigor, Prompt Optimization, and Human Curation). Taxonomy of Concordance and Divergence: Formal conceptual scheme to classify and qualify divergences between human annotations and automated inferences. Mapping of Interpretative Axes: Empirical data contrasting the model's behavior between the Literal-Descriptive Axis and the Analytical-Subjective Axis.