Graph-MARL-CloudLLM: A Graph-enhanced Multi-Agent Reinforcement Learning Framework for Adaptive Resource Scheduling and Workload Optimization of Large Language Model Inference Services across Multi-Cloud Platforms
The increasing adoption of Large Language Models (LLMs) in enterprise artificial intelligence applications has created significant challenges for inference service management across multi-cloud platforms, where heterogeneous computing resources, dynamic workloads, network fluctuations, and strict quality-of-service (QoS) requirements hinder efficient resource utilization and stable service delivery. This paper aims to develop an intelligent scheduling framework for cross-cloud LLM inference services that can achieve adaptive workload orchestration, low-latency inference, and cost-efficient resource allocation. We propose Graph-MARL-CloudLLM (Graph-enhanced Multi-Agent Reinforcement Learning for Cross-Cloud Large Language Model Inference Optimization), a novel framework that integrates Graph Neural Networks (GNNs) with Multi-Agent Reinforcement Learning (MARL) to model complex cloud resource dependencies and perform autonomous scheduling decisions. Specifically, Graph-MARL-CloudLLM constructs a dynamic resource graph containing cloud nodes, GPU clusters, LLM instances, and inference requests, and employs cooperative agents for cloud selection, model selection, GPU allocation, and request routing. A QoS-aware reinforcement learning strategy is further designed to jointly optimize latency, throughput, cost, and resource utilization. Experimental results on a simulated heterogeneous multi-cloud LLM inference environment demonstrate that Graph-MARL-CloudLLM achieves superior scheduling performance compared with existing approaches. Specifically, the proposed framework reduces P99 latency from 271 ms to 194 ms, improves GPU utilization to 86.9%, and decreases inference cost by 17.7% compared with the MARL-based baseline. These results validate the effectiveness of graph-enhanced state representation and cooperative multi-agent optimization for adaptive LLM inference scheduling.