A Comprehensive Benchmark for Graph Textual Summarization

Abstract

This paper introduces a novel task, graph textual summarization, which aims to generate natural language descriptions that capture both the semantic content and structural characteristics of graph data based on large language models (LLMs). Unlike traditional summarization tasks for text, images, or videos, summarizing graph data presents unique challenges due to its abstract and non-intuitive nature. We develop a comprehensive benchmark construction framework and construct three diverse benchmark datasets, including text-attributed graphs (TAGs), knowledge graphs (KGs), and heterogeneous attributed graphs (HAGs). To facilitate empirical studies, we deliver three representative benchmark datasets: SurveyGraphSum, consisting of 7835 citation graphs from survey papers across 10 major domains and 56 subdomains; BookGraphSum, containing 6175 knowledge graphs extracted from 186 books; and E-comGraphSum, with 912 heterogeneous graphs from real-world e-commerce networks, each representing a fraud group with 14 node types, 19 relation types, and 9 numerical attributes related to user behavior and interactions. In addition, we propose a unified evaluation framework that assesses both linguistic quality and factual consistency, providing a comprehensive approach to evaluating generated summaries for content fidelity and accuracy. By formalizing the task, releasing benchmark datasets, and proposing an evaluation framework, we aim to advance research in graph textual summarization and promote methods that produce accurate, informative, and human-readable summaries for complex graphs.