Exploring Large Language Models as Decision Support Tools: A Proof of Concept for Procedure-Related Information Retrieval

Abstract

Policy and procedural documentation are essential for the effective operation of government agencies. However, the vast volume of these documents often obscures critical information within hundreds of pages of unrelated content. Recent advancements in large language models (LLMs) enhance text search and response generation capabilities, enabling efficient extraction and summarization of key content from extensive documents. This study evaluates whether open-source LLMs can generate accurate policy briefs to support decision-making. The research has two primary objectives: (1) to assess multiple LLMs available through Ollama, including LLAMA 3.1-8B, Mistral 7B, and Gemma 2, to determine their efficacy with policy and procedural texts, and (2) to assess whether these generated responses can serve as a practical tool for government employees. Evaluation metrics include computational analyses and feedback from professionals in the public policy sector. Preliminary results indicate that open-source LLMs effectively analyze extensive policy documentation and generate concise, relevant summaries in response to user queries. We conclude that LLMs present a valuable resource for policymakers and government employees, enabling efficient access to accurate summaries that support policy development and implementation. Future work is still needed to improve response structure and explore the capabilities of larger models.