Application of large language models in legal queries: experiments with the artificial intelligence for legal assistance challenge
TL;DR
It is demonstrated that combining LLMs with semantic retrieval techniques enhances precision and scalability in legal information systems, offering a viable roadmap for developing domain-specific, efficient, and sustainable AI legal assistants.
Abstract
This study evaluates the applicability of generative artificial intelligence (AI), specifically Large Language Models (LLMs), in legal assistance tasks using the Artificial Intelligence for Legal Assistance challenge dataset, which includes 197 statutes, 2914 judicial cases, and 50 legal queries. The research compared conversational LLM tools with semantic retrieval systems based on dense embeddings. Results show that general-purpose tools (ChatGPT, NotebookLM) achieved poor discrimination, while embedding-based methods significantly improved accuracy: the OpenAI text-embedding-ada-002 model reached 46.43% retrieval accuracy, and a fine-tuned all-mpnet-base-v2 model improved from 18.9% to 31.87% (a 68.6% relative gain). These findings demonstrate that combining LLMs with semantic retrieval techniques enhances precision and scalability in legal information systems, offering a viable roadmap for developing domain-specific, efficient, and sustainable AI legal assistants.