Comparison of AI agents for creating SQL queries

Abstract

Large language models have been increasingly applied for Text-to-SQL tasks recently, due to the fact that generating correct SQL queries is the most important factor. This study presents a comparison of AI agents based on open-source and closed-source large language models for generating SQL queries. GPT-4o, Claude 3.7 Sonnet, and LLaMA 3 8B were evaluated using two widely known datasets: Spider and WikiSQL. The chosen architectures were compared using Exact Match, F1-score, BERTScore, Execution Accuracy and processing time. Obtained results show that all agents perform well on simple queries, while agents based on closed-source models achieve better performance on complex SQL generation tasks.