LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. Operating MCP at cloud scale, however, becomes difficult. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tool is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy, and reduces tool selection time by $8.9\times$ and token usage by $23.8\times$, with low per-call overhead, stable under scale-out. Finally, we share the lessons learned from deploying the gateway system in production.
A reference architecture is developed that separates proposal generation from governed execution, identifies recurring integration patterns for LLM-enabled operations, and derives a research agenda for higher network autonomy under explicit assurance, safety, and governance constraints.
Tianzhu Zhang, Changgang Zheng, Shanshan Wang et al.· 0 citations
An architecture that decouples complex policy enforcement from high-speed packet forwarding to support VPC semantics on back-end NICs and enable front-end/back-end integration is proposed, suggesting that commodity hardware can support both high-throughput AI training and flexible VPC features.
Yinhe Wang, Xing Li, Enge Song et al.· Asia-Pacific Workshop on Net...· 0 citations
ParaFlowO is proposed, an architecture that Parallelizes processing elephant Flows across multiple CPU cores while preserving in-Order delivery and integrates a lightweight reordering mechanism to preserve packet order and controls parallelism to mitigate contention on shared state.
Chang-Gang Zheng, Bowen Yang, Jin Ke et al.· Asia-Pacific Workshop on Net...· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
