R2Act, a recovery-action evaluation framework for post-diagnosis incident response, provides a reproducible, simplified starting point for research and evaluation and reveals that many recovery failures arise not from missing diagnostic knowledge, but from the difficulty of translating diagnostic evidence into valid recovery actions and admissible targets.
Abstract
Large language models (LLMs) are increasingly used to interpret operational evidence and assist incident response in cloud-native microservice systems. However, recovery-oriented use cases require more than identifying a root cause. After observing symptoms and diagnosing a fault, an operator or agent must translate the diagnosis into a concrete recovery action, apply it to an admissible target, and verify that service health has been restored. Existing RCA and log-analysis evaluations are well-suited to diagnosis, but they do not characterize this subsequent action decision. This paper presents R2Act, a recovery-action evaluation framework for post-diagnosis incident response. R2Act defines an incident schema, quality gate, action-space representation, recovery-validity metrics, offline evaluator, and live-replay protocol. We instantiate the framework as a benchmark dataset of 302 quality-audited Kubernetes incidents from \system. Each incident provides synchronized multi-modal observations, root-cause labels, an incident-specific action space, and annotated valid and invalid recovery plans. We evaluate heuristic, supervised, RCA-oriented, deep log, and LLM-based methods. The strongest RAG-based LLMs reach 91.4\%--99.7\% root-cause service accuracy, yet their recovery validity remains only 36.8\%--60.3\%. Even when both the root-cause service and fault type are correct, recovery-oriented methods still choose invalid actions for 39.5\%--62.0\% of correctly diagnosed incidents. Overall, this work reveals that many recovery failures arise not from missing diagnostic knowledge, but from the difficulty of translating diagnostic evidence into valid recovery actions and admissible targets. This work provides a reproducible, simplified starting point for research and evaluation.
RunbookFX is presented, a typed functional domain-specific language that elevates incident response from natural-language suggestions to executable programs whose safety is established statically, and key insight is that incident-response safety decomposes into three interacting dimensions: risk severity, exercised capabilities, and rollback resource availability.
Modern LLM-based applications increasingly require multi- stage execution, persistent intermediate state, retry seman- tics, and auditable usage accounting. However, many LLM applications are still implemented as stateless prompt- response wrappers or session-bounded conversational sys- tems, which makes them difficult to recover, audit, and re- produce after interruption or failure. We propose AgentR, a stateful architecture for LLM workflow systems that en- ables persistence and recovery, instantiated through scien- tific literature review as a representative use case. AgentR represents research intent, generated queries, candidate- paper assessments and gap analyses as durable workflow artifacts, and executes the pipeline through asynchronous BullMQ workers backed by Redis, with PostgreSQL as the persistence store. The design includes explicit processing state transitions, retries with exponential backoff, orphan job detection, credit-aware pre-checks, ACID token-cost logging, and Type-2 slowly changing pricing records. We evaluate AgentR on telemetry collected from a prototype deployment. At the LLM stage, the system achieves 99.2% job completion, and mean latencies of 9.0 s, 18.9 s, and 25.4 s for intent decomposition, query generation, and paper scoring, respectively. Parallel scoring allows for analytical latency modeling from observed calls, leading to as much as 4.3 wall-clock speedup over sequential execution. The results provide preliminary proof-of-concept that persistent state machine design, asynchronous orchestration, and cost-aware usage logging can enable improved observability, recoverability, and operational accountability in LLM workflow systems. The prototype implementation of AgentR is publicly available at: https://github.com/ RiyaSamanta/AgentR-public.
Riya Samanta, Bidyut Saha, S. K. Ghosh et al.· 0 citations
Microservice failures are often diagnosed from operational telemetry. However, automated program repair systems usually start from issue reports, localized code context, or failing tests. This mismatch leaves a gap between telemetry-based diagnosis and patch generation. We present ORCA, an observability-grounded APR pipeline for microservice incidents. ORCA first distills the differences in paired failure and reference telemetry into a fault signature, then uses the signature to identify candidate code and deployment-configuration locations. Repair graph agents and an Exploration agent generate unified-diff patch candidates from these locations. ORCA evaluates generated patches with a Telemetry-Grounded Patch Verifier that separates patch validity, syntactic and semantic correctness, test-oracle integrity, and telemetry replay. On a 575-case benchmark, ORCA outperforms all evaluated baselines in terms of cost-effectiveness. Results show that operational telemetry can be transformed from diagnostic evidence into actionable repair context: paired telemetry supports repair-oriented localization, while repair graph agents convert localized code and configuration evidence into constrained patch-generation context for the LLM. Telemetry-grounded verification then exposes repair outcomes that issue- or test-only evaluation would miss.
Yuanchen Gao, Yifang Tian, Yiran Li et al.· 0 citations
RedAgentBench is introduced, an executable framework for autonomous red-teaming and faithful measurement that shows that executable evaluation can improve safety measurement and identify actionable intervention points.
Zixing Chen, Xingyuan Liu, Jie Zhu et al.· 1 citation
: Enterprise-scale distributed microservices in regulated environments operate under stringent availability, latency, and auditability requirements. Traditional monitoring approaches detect anomalies reactively, after service degradation has already impacted end users or regulatory SLA (Service Level Agreement) obligations. This paper presents the Cognitive Agentic Self-Healing Platform (CASP): an LLM-orchestrated agentic system that continuously monitors distributed microservice environments, autonomously diagnoses fault conditions across six failure categories, and executes targeted remediation actions without human intervention. CASP integrates a structured four-phase autonomous loop comprising detection, diagnosis, remediation, and verification, with an LLM decision engine supporting both cloud-hosted and locally deployable backends, enabling deployment in air-gapped environments subject to data residency requirements. Evaluation on a controlled synthetic workload, calibrated against published benchmark profiles, demonstrates 88.3% overall diagnostic accuracy, an 83% reduction in mean recovery time compared to no self-healing, and a 53% reduction compared to static rule-based approaches. Post-recovery latency stabilizes within 7 minutes. The locally deployable LLM backend achieves p95 decision latency of 4.9 seconds, within the 5-second operational SLA threshold, making it viable for production deployment in regulated environments subject to data residency requirements.
Ketankumar Savajiyani· International Conference on...· 0 citations
Large language models (LLMs) can turn a flood of cross-layer industrial logs into a fluent incident narrative, but a narrative that cites only real, resolvable events can still be wrong in every relation that matters: the login came from a different workstation, the write command occurred after the physical change it supposedly caused, the action fell inside a planned maintenance window, and the controller does not even actuate the affected process. A cited event is not necessarily supporting evidence. When such a narrative drives automated response, the error propagates into isolating the wrong controller or revoking a legitimate operator. We present EviGuard, a system that decides when an LLM’s understanding is trustworthy enough to act on. EviGuard stores auditable cross-layer evidence in a provenance graph, lets the LLM propose only hypotheses, compiles each hypothesis into atomic machine-checkable claims in an Incident Claim Language, and has an ensemble of deterministic verifiers label every claim supported, contradicted, or unknown against the graph—honoring interval time, event-time policy and credential versions, network reachability, and physical control dependencies. A response gate forbids any high-impact action whose critical preconditions are not all supported. On EviCPS-Bench (42 hardware-in-the-loop attack chains, 9600 claim-level labels, κ=0.87), EviGuard cuts the unsupported-claim rate from 12.6% to 1.7%, raises relation-edge F1 from 0.64 to 0.89, holds prompt-injection success to 0.4%, and executes zero unverified high-impact actions across 3200 response decisions, at a median end-to-end latency of 0.44 s.
Hao-Zhe Zhou, Hang Lei, Mao-Lin Yang· Applied Sciences· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
