Graph-MARL-CloudLLM: A Graph-enhanced Multi-Agent Reinforcement Learning Framework for Adaptive Resource Scheduling and Workload Optimization of Large Language Model Inference Services across Multi-Cloud Platforms
TL;DR
Experimental results on a simulated heterogeneous multi-cloud LLM inference environment demonstrate that Graph-MARL-CloudLLM achieves superior scheduling performance compared with existing approaches, and validate the effectiveness of graph-enhanced state representation and cooperative multi-agent optimization for adaptive LLM inference scheduling.
Abstract
The increasing adoption of Large Language Models (LLMs) in enterprise artificial intelligence applications has created significant challenges for inference service management across multi-cloud platforms, where heterogeneous computing resources, dynamic workloads, network fluctuations, and strict quality-of-service (QoS) requirements hinder efficient resource utilization and stable service delivery. This paper aims to develop an intelligent scheduling framework for cross-cloud LLM inference services that can achieve adaptive workload orchestration, low-latency inference, and cost-efficient resource allocation. We propose Graph-MARL-CloudLLM (Graph-enhanced Multi-Agent Reinforcement Learning for Cross-Cloud Large Language Model Inference Optimization), a novel framework that integrates Graph Neural Networks (GNNs) with Multi-Agent Reinforcement Learning (MARL) to model complex cloud resource dependencies and perform autonomous scheduling decisions. Specifically, Graph-MARL-CloudLLM constructs a dynamic resource graph containing cloud nodes, GPU clusters, LLM instances, and inference requests, and employs cooperative agents for cloud selection, model selection, GPU allocation, and request routing. A QoS-aware reinforcement learning strategy is further designed to jointly optimize latency, throughput, cost, and resource utilization. Experimental results on a simulated heterogeneous multi-cloud LLM inference environment demonstrate that Graph-MARL-CloudLLM achieves superior scheduling performance compared with existing approaches. Specifically, the proposed framework reduces P99 latency from 271 ms to 194 ms, improves GPU utilization to 86.9%, and decreases inference cost by 17.7% compared with the MARL-based baseline. These results validate the effectiveness of graph-enhanced state representation and cooperative multi-agent optimization for adaptive LLM inference scheduling.