ViNL2Vis-FaithBench: A Vietnamese–English Benchmark and Faithfulness Evaluator for Natural-Language-to-Visualization

Abstract

Natural-language-to-visualization (NL2Vis) systems are evaluated by whether they produce a valid specification or a rendered chart, although these signals do not imply that the chart is grounded in the user’s request or the data. This mismatch is problematic for multilingual data analysis, where a system must decide not only how to plot but also when to clarify or refuse. We introduce ViNL2Vis-FaithBench, a Vietnamese-English benchmark built from 783 tabular datasets and 33,700 synthetic benchmark samples, together with a deterministic evaluator for data faithfulness. The central idea is to treat chart generation as a set of testable commitments rather than as a rendering event. The evaluator decomposes outputs into execution, schema grounding, transformation behavior, visualization choice, and plot/clarify/refuse decisions, exposing failures hidden by render-only metrics. Across local Ollama Qwen3-4B strict-transform pipeline artifacts, faithfulness reaches 95.025 on a Vietnamese balanced split, 95.316 on an English explicit split, and 95.377 on matched bilingual explicit pairs; the frozen Vietnamese held-out split reaches 96.157 with the local Ollama Qwen3-8B strict-transform pipeline. On the 300-sample Vietnamese schema-diversity stress split, five local Ollama model families range from 78.502 to 91.198 faithfulness, with failures concentrated in chart-family ambiguity and render formatting rather than unsafe plotting. These are pipeline scores, not raw LLM-only scores. Ablations show that stricter prompting, deterministic repair, and audit-driven transformation checks reduce chart-family, rendering, and transformation failures that raw model outputs leave unresolved. The benchmark is best interpreted as a controlled faithfulness evaluation over many real indicator tables, not as evidence for unrestricted generalization to all tabular visualization settings.