From Clinical Free Text to Auditable Concepts: An Agentic Framework for Interpretable Prediction

Abstract

Across application domains, predictive signals often sit in unstructured free text rather than structured fields, yet turning that text into useful and interpretable features is difficult. Running large language models (LLMs) over an entire corpus is costly and hard to reproduce, while end-to-end text representations can rely on surface cues that are difficult to inspect. We present an agentic workflow that takes a prediction task and a raw text corpus as input and produces an auditable feature layer. The first two agents use an LLM to derive a task-specific predictor taxonomy and weakly label a bounded text sample; routed local extractors then process the corpus, and a deterministic builder aggregates the evidence into a dynamic, longitudinal concept bottleneck. We evaluate the framework on medication discontinuation in a longitudinal oncology cohort and 30-day readmission in MIMIC-IV. With gradient boosting, the longitudinal bottleneck increases area under the receiver operating characteristic curve (AUROC) over coarse concept buckets from 0.700 to 0.761 for medication discontinuation and from 0.576 to 0.609 for readmission. The proposed framework achieves predictive performance comparable to direct BioClinicalBERT prediction on both tasks while additionally providing explicit, interpretable, and traceable task-specific concepts. LLM use is confined to a bounded weak-labeling stage costing $24.00 and $23.39, respectively, compared with projected costs of $10,648 and $11,519 for exhaustive sentence-level LLM processing of the full corpora, demonstrating the substantial cost efficiency of the proposed agentic system.