Jul 2026· De Computis· Vol 15, pp. 439· 0 citations· 27 references
Computer Science
TL;DR
A language-independent benchmarking framework for evaluating fault tolerance in actor-based runtimes, which characterises supervised crash–recovery behaviour for largely stateless actor services rather than providing a comprehensive evaluation of actor-based fault tolerance.
Abstract
Fault tolerance is a fundamental requirement of distributed systems, and actor-based runtimes provide a widely adopted approach for building resilient and highly concurrent applications. Although several actor ecosystems offer mechanisms for supervision, failure detection, and recovery, comparative studies frequently focus on performance metrics rather than fault-tolerance behaviour. This paper presents a language-independent benchmarking framework for evaluating fault tolerance in actor-based runtimes. The framework was implemented using three representative ecosystems: Elixir/BEAM, Scala/Akka, and Go/Proto.Actor. A distributed chat-based benchmark application was used to measure throughput, reconnection latency, and failure-detection latency under recurring transient failures. All implementations followed an equivalent architecture and were executed under identical experimental conditions. The study deliberately targets a single, well-defined fault model: the supervised crash recovery of in-memory, effectively stateless actor services, in which chat actors are abruptly terminated and restarted by their supervisors while clients rediscover and reconnect to them. Stateful recovery (actor state, mailbox contents, in-flight or persistent messages), as well as multi-node network effects, are explicitly out of scope. Accordingly, the benchmark characterises supervised crash–recovery behaviour for largely stateless actor services rather than providing a comprehensive evaluation of actor-based fault tolerance. The results reveal distinct trade-offs among the evaluated ecosystems. Elixir achieved the highest throughput and the lowest throughput variability under fault conditions, while Scala/Akka consistently provided the lowest reconnection and failure-detection latencies, particularly at large scale. Go/Proto.Actor remained competitive in throughput-oriented scenarios but showed greater degradation in recovery-related metrics as concurrency increased. The results indicate that no single runtime dominates all evaluated dimensions of recovery behaviour. Beyond the runtime comparison, this work contributes a reproducible benchmarking framework that provides a foundation for future empirical studies of actor-based runtime recovery under controlled fault conditions.
Fault injection is widely used to evaluate the resilience of microservice systems, where client requests often span multiple services and execution stages. Existing request-level techniques usually control where and what faults are injected, but not when they are activated within a distributed execution. This limitation makes it difficult to reproduce timing-dependent failures, such as failures after state-changing side effects, order-sensitive concurrent responses, and partial failures among repeated downstream calls. This paper presents SequenceFI, a non-intrusive framework for temporal fault injection in microservice systems. SequenceFI observes message-level send and receive events, propagates compact temporal evidence along request executions, and triggers faults only when occurrence-sensitive temporal guards are satisfied. It further synthesizes temporal guards from traces, reducing the need for exhaustive enumeration of temporal fault-injection configurations, while requiring no modifications to application code or serialization libraries. We implement SequenceFI on Kubernetes and evaluate it on four widely used microservice benchmarks. Across nine temporal-fault scenarios and 450 valid trials, SequenceFI achieves 100.0\% temporal success without premature or multiple injections, finds effective configurations in one attempt on average, and reduces aggregate end-to-end search time by 95.91\% compared with H-Random.
Yuzhen Tan, Jian Wang, Bing Li et al.· arXiv.org· 0 citations
Distributed in-network programs are increasingly deployed in data centers for their performance benefits, but shifting application logic to switches also enlarges the failure domain. Ensuring their correctness before deployment is thus critical for reliability. While prior verification frameworks can efficiently verify programs running on a single switch, they overlook the common interactive behaviors in distributed settings, thereby missing related bugs that can cause system failures. This paper presents Procurator, a verification framework that efficiently captures interactive behaviors in distributed in-network programs. Procurator models each P4 pipeline as a reactive actor and unifies their interactions as message passing to capture interactive behaviors under an event-driven paradigm. To improve the verification efficiency, Procurator employs an intermediate representation (IR) pruner to reduce the execution space and a schedule-replay-based acceleration approach to avoid explicit exploration of long execution traces. Evaluation shows that Procurator uncovers 28 distinct bugs in twelve real-world distributed in-network systems, and achieves up to a 9.1X speedup over the state-of-the-art framework.
Mingyuan Song, Huan-Xing Shen, Jinghui Jiang et al.· Conference on Applications,...· 0 citations
: Enterprise-scale distributed microservices in regulated environments operate under stringent availability, latency, and auditability requirements. Traditional monitoring approaches detect anomalies reactively, after service degradation has already impacted end users or regulatory SLA (Service Level Agreement) obligations. This paper presents the Cognitive Agentic Self-Healing Platform (CASP): an LLM-orchestrated agentic system that continuously monitors distributed microservice environments, autonomously diagnoses fault conditions across six failure categories, and executes targeted remediation actions without human intervention. CASP integrates a structured four-phase autonomous loop comprising detection, diagnosis, remediation, and verification, with an LLM decision engine supporting both cloud-hosted and locally deployable backends, enabling deployment in air-gapped environments subject to data residency requirements. Evaluation on a controlled synthetic workload, calibrated against published benchmark profiles, demonstrates 88.3% overall diagnostic accuracy, an 83% reduction in mean recovery time compared to no self-healing, and a 53% reduction compared to static rule-based approaches. Post-recovery latency stabilizes within 7 minutes. The locally deployable LLM backend achieves p95 decision latency of 4.9 seconds, within the 5-second operational SLA threshold, making it viable for production deployment in regulated environments subject to data residency requirements.
Ketankumar Savajiyani· International Conference on...· 0 citations
Agent systems rely on LLM APIs for every response, but these APIs can return server errors, truncated responses, or corrupted content that propagates through downstream agents and causes task failure. Evaluating robustness under these faults is crucial for reliable deployment. Existing fault injection methods are offline, require source code modification, or cannot modify specific response fields. A comprehensive evaluation also requires a systematic fault taxonomy because different fault types affect downstream agents differently. We propose AgentChaos, a chaos engineering framework for controlled, runtime, non-intrusive LLM API fault injection. Since all agent systems access LLMs through the same HTTP interface, we inject faults at this shared layer without modifying source code. We define crash, omission, and value faults on content and tool call fields, intercept and modify LLM API responses at runtime, and verify whether each fault is triggered to filter untriggered tasks and avoid underestimating fault impact. Evaluations across agent systems, benchmarks, and backbone LLMs under 65 fault configurations show that all systems degrade under fault injection, with pass@1 dropping by up to 50 percentage points. The ranking is consistent across models, suggesting that robustness depends on system implementation rather than model capability. Existing fault diagnosis methods achieve below 53% accuracy on fault type and below 56% on fault step, leaving room for improvement. We further reveal practical findings for agent system developers.
Gou Tan, Zhensu Sun, Jieke Shi et al.· 0 citations
AGENTCHAOSBENCH is presented, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry, and its held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods.
Chenkai Zhang, Yiran Li, Yifang Tian et al.· 0 citations
DBA-Bench is presented, a benchmark addressing four gaps between evaluation and production operations: live-environment fidelity, outcome-first evaluation, and controlled scenario reproducibility, which uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations.
