Empirical benchmarking of large language models for data science coding: a multidimensional evaluation
TL;DR
The LLM4DS-Benchmark is introduced and a multidimensional empirical evaluation of seven large language models is conducted, highlighting the need for multidimensional, task-aware benchmarking and suggesting that model selection for data science coding should be guided by task characteristics and practical constraints rather than aggregate success rate alone.
Abstract
Despite growing enthusiasm for large language models (LLMs) as coding assistants, there remains limited empirical evidence of their effectiveness in domain-specific contexts such as data science. Existing benchmarks primarily focus on general-purpose programming and do not fully capture the challenges of data science tasks, which require data manipulation, statistical reasoning, algorithmic problem solving, and visualization. They also rarely assess practical dimensions such as first-attempt reliability, output consistency, error recovery, and cost efficiency. To address this gap, we introduce the LLM4DS-Benchmark and conduct a multidimensional empirical evaluation of seven LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, o3-mini, GPT-4.1, GPT-4o, Qwen3-Coder, and Perplexity Sonar—on 814 Python data science coding problems from StrataScratch platform, spanning Analytical, Algorithm, and Visualization tasks across three difficulty levels. Each problem received up to three attempts under a branching protocol that separates independent attempts from feedback-guided retries, enabling analysis of correctness, Pass@1, retry recovery, output consistency, execution behavior, visualization quality, code similarity, token usage, and cost per solved problem. Results show that Gemini 2.5 Pro achieved the highest overall success rate (81.3%) and Pass@1 (62.2%), but at a median cost of \$0.10740 per solved problem—316 times higher than Qwen3-Coder (\$0.00034). Across models, retries improved performance by 19–24 percentage points, with feedback resolving 20–31% of initial failures. Output consistency varied significantly across identical prompts, particularly for Analytical tasks. Model rankings also shifted by task type and evaluation dimension, with no single model dominating across all categories. Instead, a Pareto-optimal set—Qwen3-Coder, GPT-4.1, o3-mini, Claude Sonnet 4.5, and Gemini 2.5 Pro—emerged, reflecting trade-offs among accuracy, cost, and reliability. These findings highlight the need for multidimensional, task-aware benchmarking and suggest that model selection for data science coding should be guided by task characteristics and practical constraints rather than aggregate success rate alone.