Skip to content
#software testing Review Open access

Proof of Concept of Large Language Models for Opioid Treatment Policy Surveillance: 97% Agreement With Subject Matter Experts.

Aug 2026 · Journal of Studies on Alcohol and Drugs · 0 citations
Medicine

Abstract

Objective

Policy surveillance typically involves detailed, time-consuming manual screening of policies for inclusion in a final dataset. This screening process involves risks of human error and inconsistent application of inclusion/exclusion criteria, especially in complicated legal landscapes like the US opioid treatment landscape. Large language models (LLMs) could assist human subject matter experts (SMEs) during screening, but LLMs have been understudied for policy surveillance. Therefore, we conducted a test comparing opioid treatment policy screening decisions between SMEs and an LLM.

Methods

Using a Boolean search string in legal software, we identified 99 potentially relevant Massachusetts policies for emergency department opioid addiction treatment, and we downloaded text from government websites. Next, we compared two approaches to screening those policies using pre-defined inclusion and exclusion criteria: (a) manual screening by three SMEs, and (b) an LLM approach. We assessed the overall percentage of inclusion/exclusion decisions where the LLM made the same decision as the SMEs. We also identified the percentage of policies selected for inclusion by the SMEs with which the LLM agreed and potential reasons for discrepancies.

Results

The LLM made the same decision for 96 of 99 policies (97% of the time). All policies that SMEs chose to include (n=2) were also included by the LLM. Discrepancies reflected implicit inclusion and exclusion criteria used by SMEs but not provided to LLMs.

Conclusion

LLMs could serve as a quality control check during opioid policy surveillance research, supplementing human review. The policy surveillance field would benefit from best practices and technical guidelines for LLM utilization.

Read PDF

Similar papers

#software testing Review Aug 2026

Model-Based Agentic Software Engineering

MAGE explains how externalized knowledge, bounded action, independent evaluation, and retained human authority can compose into a governed engineering environment, and proposes tests of when that environment turns commodity intelligence into durable engineering progress.

James C. Davis, Kelechi G. Kalu, Huiyun Peng et al. · 1 citation
#software testing Preprint Aug 2026

Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code

LLMs are increasingly used for code generation, yet they frequently hallucinate non-existent software packages, creating exploitable entry points into the software supply chain. We make four contributions to this problem. First, we show that prior evaluation methodologies systematically inflate hallucination rates by misclassifying standard-library modules as hallucinations in some languages. For Python, the overestimation reaches 9.4 percentage points. Second, we evaluate seven inference-time defenses for mitigating package hallucinations, including five guided decoding strategies (Greedy, Contrastive, DoLa, Nudging, and Active Layer-Contrastive Decoding), an iterative self-refinement approach (Self-Refine), and a Retrieval-Augmented Generation (RAG)-based defense.. Across eight models spanning five families and four programming languages (Python, JavaScript, Ruby, Rust), RAG reduces the package hallucination rate (PHR) in 18 of 32 model--language configurations. Third, we introduce Package Utility (PU) to assess whether defenses preserve valid and task-relevant recommendations. Among strategies evaluated, Greedy decoding provides the strongest average mitigation--utility trade-off. Fourth, we stress-test all strategies under adversarial prompts seeded with fabricated package names and find that PHR surges by up to 45 percentage points relative to standard prompts, with Ruby consistently the most vulnerable language (80.9--95.2\%). Under adversarial conditions, RAG and Self-Refine outperform all decoding-only strategies, indicating that robust defense requires either external grounding or iterative self-verification when prompts are actively hostile. Our results recast package hallucination as both a measurement problem and a decoding-time control problem, and they demonstrate that the choice of defense must be matched to the threat model and recommendation utility.

Albérick Euraste Djiré, Iyiola E. Olatunji, Melissa Tessa et al. · 1 citation
#software testing Open access Aug 2026

HawkEye: Web Vulnerability Analysis and Security Audit Tool

HawkEye is introduced, a modular, web-based vulnerability auditing platform designed to streamline security analysis by integrating multiple scanning tools within a unified dashboard and illustrates how consolidated reporting improves vulnerability prioritization for development teams.

D. R. Patil, Varad Salgare, Devaj Arya et al. · 0 citations
#software testing Review Open access Sep 2026

Application for Tracking Bugs in Software Development: Improving Project Management Efficiency

By streamlining workflows and fostering collaboration, this platform offers a scalable, cost- effective solution for SMEs and contributes to software engineering by demonstrating how integrated technologies can modernize development processes in resource limited contexts, with potential for broader adoption in Albania and beyond.

Griselda Belba TRUPJA, Agim Kasaj · 0 citations
#software testing Open access Sep 2026

Prevalence of non-union complication in femoral shaft fracture patients who underwent IM nailing surgery in Fatemi Hospital Ardabil.

The findings of the present study indicated that potential complications such as delayed union, nonunion, and osteomyelitis in the intramedullary nailing method are approximately comparable to those of the external fixator method.

Reza Noktesanj, Ali Nami, F. Amani et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Aug 17, 2026

Q&A: Rethinking how innovation happens

In his latest book, Professor Eugene Fitzgerald examines the forces that turn breakthroughs into value — and why innovation resists simple formulas.