KVQuake: Fault Injection and Recovery Assessment for Disaggregated Large Language Model Serving

Abstract

Disaggregated large language model serving introduces a recovery problem that spans request execution, remote key-value (KV) cache reuse, and service lifecycle management. A successful response does not by itself establish that the remote cache path has recovered. This report describes KVQuake, an external fault-injection and evidence collection tool for a pinned vLLM-LMCache deployment. The tool supports six fault scenarios, correlates connector actions with request execution, and records experiment validity, observed system behavior, and cleanup as separate outcomes. We document historical negative results and subsequent investigations without combining runs from different source revisions into a performance comparison. We also report a fixed-runtime F6 case study with three independent service startups and paired healthy, request-only recovery, and full-repair phases. All 63 workload requests returned HTTP 200 with token sequences equal to their within-trial healthy references. During request-only recovery, all 21 requests used local fallback while the gateway circuit remained open. In every trial, an intermediate LMCache restart passed health checks but failed the actual remote-cache-hit probe; the existing repair path subsequently rebuilt the owned data plane and restored a 16-token remote hit. Full repair actions took 98.28-99.13 seconds. These observations support a layered recovery assessment in the tested configuration. The accompanying artifact package preserves source identities, negative and unresolved outcomes, redacted evidence, and offline analysis. The report does not estimate production failure probabilities, claim throughput improvements, or establish cross-model generality. Licensing: Report and author-owned experimental data: CC BY 4.0. Author-owned accompanying software, verification scripts and source inspection archive: MIT. See NOTICE.md and LICENSES/ in the companion archive for component scopes and third-party exclusions. Independent formal repetitions: three service startups on one host, 63 workload requests. Historical examples span different code versions and are not a controlled performance comparison. This technical report has not received independent external peer review.