Proof of Concept of Large Language Models for Opioid Treatment Policy Surveillance: 97% Agreement With Subject Matter Experts.

Objective

Policy surveillance typically involves detailed, time-consuming manual screening of policies for inclusion in a final dataset. This screening process involves risks of human error and inconsistent application of inclusion/exclusion criteria, especially in complicated legal landscapes like the US opioid treatment landscape. Large language models (LLMs) could assist human subject matter experts (SMEs) during screening, but LLMs have been understudied for policy surveillance. Therefore, we conducted a test comparing opioid treatment policy screening decisions between SMEs and an LLM.

Methods

Using a Boolean search string in legal software, we identified 99 potentially relevant Massachusetts policies for emergency department opioid addiction treatment, and we downloaded text from government websites. Next, we compared two approaches to screening those policies using pre-defined inclusion and exclusion criteria: (a) manual screening by three SMEs, and (b) an LLM approach. We assessed the overall percentage of inclusion/exclusion decisions where the LLM made the same decision as the SMEs. We also identified the percentage of policies selected for inclusion by the SMEs with which the LLM agreed and potential reasons for discrepancies.

Results

The LLM made the same decision for 96 of 99 policies (97% of the time). All policies that SMEs chose to include (n=2) were also included by the LLM. Discrepancies reflected implicit inclusion and exclusion criteria used by SMEs but not provided to LLMs.

Conclusion

LLMs could serve as a quality control check during opioid policy surveillance research, supplementing human review. The policy surveillance field would benefit from best practices and technical guidelines for LLM utilization.