NERFlow: A Workflow-Based Subsystem of FIT4NER for LLM-Assisted Medical Named Entity Recognition
TL;DR
NERFlow is introduced, a workflow-driven subsystem of the FIT4NER project whose contribution is an abstraction layer that makes rule-based, model-based, and LLM-based annotation interchangeable and comparable within one configurable workflow environment.
Abstract
Preparing training data for domain-specific medical Named Entity Recognition (NER) involves a trade-off between annotation quality, expert effort, and data privacy: manual annotation is costly, whereas cloud-based Large Language Models (LLMs) raise concerns about the control of sensitive clinical text. This article introduces NERFlow, a workflow-driven subsystem of the FIT4NER project whose contribution is an abstraction layer that makes rule-based, model-based, and LLM-based annotation interchangeable and comparable within one configurable workflow environment. Open-source LLMs are integrated as exchangeable annotation services, deployable locally or in cloud-agnostic infrastructures via Kubernetes, and embedded into the KM-EP knowledge management system. NERFlow was evaluated qualitatively, through a cognitive walkthrough, an IEEE 1028 technical review, and a user-centered survey with 18 participants, and quantitatively on the CRAFT corpus with seven open-source and hosted LLMs run through an identical pipeline. The results support its use as LLM-assisted pre-annotation with expert correction, with locally deployable open-source models as the more reliable basis for reproducible operation.