A novel Continual Learning Dynamic Graph LLM framework (Continual-GraphLLM) is proposed to continually adapt to incoming patterns by routing them to experts specialized in similar past patterns, while mitigating the overwriting of previously learned patterns by assigning new experts to unseen patterns.
Tianhang Wan, Xin Wang, Haibo Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Text-guided molecule generation enables controlled molecular design from natural language descriptions and has broad applications in areas such as drug discovery. While recent methods have demonstrated promising capability in generating molecules that align well with textual descriptions, they often overlook the structural properties of the generated graphs. As a result, these approaches struggle to simultaneously ensure consistency with the input text and high structural quality of the generated molecules. In this paper, we propose a text-guided molecular graph generation framework that leverages the structural modeling power of graph diffusion models to achieve both strong alignment with textual descriptions and high-quality molecular structures. However, accomplishing this goal involves several key challenges: 1) how to align graph diffusion models with natural language instructions in order to generate molecular graphs with expected relational semantics from text, 2) how to directly optimize the quality of the generated molecular graphs without sacrificing fine-grained alignment with text-specific details. To tackle these challenges, we introduce Text-guided Conditional Discrete Graph Diffusion (TDGD), a discrete diffusion-based framework for generating molecular graphs from natural language descriptions. Our model incorporates a structure-aware cross-attention mechanism that aligns textual semantics with molecular structures by capturing relational semantics between textual descriptions and molecular structures. In addition, we propose a molecule structure consistency loss that explicitly enforces structural coherence during generation, leading to higher-quality and more consistent molecular graphs. Extensive experiments on ChEBI-20 and L+M-24 datasets demonstrate the effectiveness of our proposed TDGD model.
Yang Yao, Xin Wang, Yaofei Wu et al.· Proceedings of the 32nd ACM...· 0 citations
This tutorial presents a comprehensive overview of three emerging and synergistic directions for tackling distribution shifts in graph learning, which highlight Graph LLMs, which combine the representational power of large language models with graph structures to enable flexible, in-context, and few-shot learning on graphs.
Xin Wang, Haoyang Li, Haibo Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Discovering optimal graph neural network (GNN) architectures for various tasks is both labor-intensive and time-consuming. To reduce human effort, graph neural architecture search (GNAS) has recently been utilized to automatically identify effective GNN architectures for specific tasks, achieving competitive or even superior performance compared to manually designed architectures. However, existing GNAS methods fail to identify optimal architectures in the presence of structural and semantic noise, where structural noise refers to missing or redundant edges within the graph structure, and semantic noise denotes inaccurate node representations derived from ambiguous node features such as textual vagueness or semantic ambiguity. In this paper, we address this problem for the first time via theoretical analyses and empirical evaluations. We discover that existing differentiable GNAS methods typically select architectures based on task-relevant information hidden in the graph, being highly sensitive to structural and semantic noise, which results in suboptimal selection of GNN architectures under noise. To handle the structural and semantic noise, we propose Curriculum-GraphLLM, a novel graphLLM framework for joint optimization of architectures, structures, and texts for denoised graph neural architecture search. The core idea is to jointly optimize GNN architectures, graph structures, and textual semantics as a unified denoising process during architecture search. Specifically, we first develop a dynamic topology updating mechanism to adaptively adjust the graph structure. Then, we jointly optimize the GNN architecture and graph structure through a curriculum-based iterative updating approach. To further deal with semantic noise on text-attributed graphs (TAGs), we introduce LLMs as an auxiliary text modeling module to refine textual semantics and guide the co-optimization of text representations, graph structures, and GNN architectures. We conduct extensive experiments to show that our proposed Curriculum-GraphLLM achieves consistently competitive or superior performance compared with existing baselines, especially under structural and semantic noise.
Xin Wang, Haibo Chen, Linxin Xiao et al.· IEEE Transactions on Pattern...· 0 citations
Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks. However, long-video scenarios remain challenging due to the tension between limited visual token budgets and the need to capture multiple key events. Existing approaches typically process long videos in two stages, i.e., i) select keyframes and ii) perform detailed perception, which exhibit limitations: they lack a modular mechanism for adaptive capacity allocation and self-correction, resulting in unreliable modeling. To tackle these challenges, we propose MoD-VLLM, a novel Modularized Dynamic-Granularity Video LLM framework for multi-event long video understanding, which unifies temporal grounding and semantic understanding iteratively and self-reflectively. Specifically, we propose a Positive-Negative Video Segments Grounding module and a Modularized Dynamic-Granularity Reflection module, which form a closed loop to progressively localize the question-related video segments. The grounding module instructs a Video LLM to distinguish relevant from irrelevant video segments based on the video question. The reflection module employs a modularized scheduler that dynamically selects fine-grained encoding for relevant positive segments to capture detailed perception and coarse-grained encoding for negative segments to maintain global context. We further propose a dynamic-granularity reinforcement learning strategy, allowing MoD-VLLM to learn optimal grounding policies and dynamic granularity visual representation jointly. Moreover, we propose MEventBench, a challenging Multi-Event Long Video Benchmark for complex long video reasoning. Extensive experiments on several long video understanding benchmarks and our MEventBench demonstrate that MoD-VLLM significantly outperforms state-of-the-art baselines.
Wei Feng, Xin Wang, Yuwei Zhan et al.· 0 citations
Dynamic text-attributed graphs (DyTAGs) exhibit coupled textual and structural dynamics, and existing mainstream approaches for DyTAGs extend conventional large language models (LLMs) to capture both dynamics, thereby giving rise to dynamic graph LLMs. However, in DyTAGs, the continuous emergence of new nodes and edges with incoming textual content and interactions drives the joint evolution of graph structural-textual patterns, causing existing methods to struggle with evolving patterns. This motivates a largely unexplored problem of continual learning on DyTAGs, which aims to adapt to constantly evolving graph structural-textual patterns while retaining past knowledge, which imposes two challenges: 1) unlike common graphs, graph structure and textual semantics in emerging DyTAG patterns jointly evolve, requiring dynamic graph LLMs to adapt structure, text, and graph-text fusion simultaneously; and 2) updating dynamic graph LLMs to fit a new pattern may destroy the global graph-text fusion capabilities and bias the model towards recent local dynamics. To address these challenges, we propose a novel Continual Learning Dynamic Graph LLM framework (Continual-GraphLLM) to continually adapt to incoming patterns by routing them to experts specialized in similar past patterns, while mitigating the overwriting of previously learned patterns by assigning new experts to unseen patterns. Specifically, we propose a graph-text factor-based router to adapt to incoming structural-textual joint patterns by utilizing latent factors to adaptively activate suitable experts. Furthermore, we design invariance regularized multi-scale experts that mitigate forgetting by capturing the invariances among learned patterns assigned to the same expert, where each expert progressively integrates structural and textual information from local scale to global scale. Extensive experiments on real-world DyTAGs demonstrate the superiority of our method over competitive baselines, highlighting its effectiveness in adapting to emerging DyTAG patterns.
Tianhang Wan, Xin Wang, Haibo Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Graph machine learning has witnessed rapid progress across both academia and industry. However, most existing methods are developed under the in-distribution (I.D.) hypothesis, which assumes that training and testing graph data are drawn from the same distribution. In real-world applications—ranging from dynamic knowledge graphs to evolving biomedical networks—this assumption is frequently violated, resulting in severe performance degradation under distribution shifts. Addressing this challenge has become a key focus in recent years, leading to the development of novel paradigms that move beyond the I.D. setting. This tutorial presents a comprehensive overview of three emerging and synergistic directions for tackling distribution shifts in graph learning. First, we highlight Graph LLMs, which combine the representational power of large language models with graph structures to enable flexible, in-context, and few-shot learning on graphs. Second, we introduce adaptation techniques for both GNNs and Graph LLMs, including graph neural architecture search and continual learning strategies for evolving data. Third, we cover generalization methods that incorporate causality and invariance principles to build robust graph models under unseen distributions. We will advocate novel, high-quality research findings, as well as innovative solutions to the challenging problems in graph machine learning under distribution shifts and the applications on graphs. This topic is at the core of the scope of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, and is attractive to machine learning as well as data mining audience from both academia and industry.
Xin Wang, Haoyang Li, Haibo Chen et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
