A Standardized Framework for Evaluating Language Quality of Chinese Large Language Models

Abstract

Chinese large language models (CLLMs) are rapidly transitioning from research to deployed infrastructure across multiple sectors. Yet current evaluation practice remains fragmented and benchmark-centric, conflating language quality with general capability and weakening comparability across models. This is especially critical in standards-oriented contexts, where language quality must be treated as a multidimensional construct encompassing linguistic correctness, semantic adequacy, discourse coherence, style appropriateness, factual grounding, safety compliance, and robustness. In this study, we propose a standardized framework for evaluating CLLM language quality as a pre-standardization, which integrates five core elements, specifically a layered quality model, scenario-driven test specification, multi-source evidence collection (automated metrics, expert review, calibrated LLM-as-a-judge), transparent scoring and conformity schemes, and governance structures aligned with AI standards practice. Unlike conventional approaches, the framework explicitly separates language quality from broader capability, introduces a hierarchical indicator system tailored to Chinese linguistic phenomena, and defines a reproducible evaluation workflow with quality assurance and version control. To operationalize the proposal, we also develop a reference architecture for quality dimensions, indicators, scoring logic, reporting structure, and pre-standard deliverables. It further demonstrates how the framework can support both research benchmarking and practical deployment scenarios, including enterprise acceptance testing and sector-specific profile extension. The resulting framework provides a technically grounded and standards-oriented blueprint for CLLM language-quality evaluation, with potential value as both a de facto industrial evaluation specification and a foundation for future formal standardization in the information and communication technology (ICT) and artificial intelligence (AI) quality ecosystem.