Log parsing, which transforms unstructured log messages into structured formats, is a critical step in automated log analysis and directly impacts the effectiveness of downstream tasks. However, existing log parsers struggle to balance effectiveness with efficiency and show limited capability in handling log inconsistency, a common phenomenon in real-world systems where log messages follow different templates but contain semantically equivalent parameters. To address these challenges, we propose SLOPE, a fine-grained log parser combining syntax with semantics. Unlike coarse-grained template-level parsing and caching, SLOPE distills parameter-level semantics into reusable rules. Specifically, it first utilizes Large Language Models (LLMs) to extract parameters and their semantic and syntactic features from sampled logs, and stores these features in a Finite State Machine (FSM). During online parsing, SLOPE reuses these semantic-syntax rules for efficient and fine-grained parsing without relying on LLMs. We conduct comprehensive evaluations on 14 public datasets and a large-scale industrial dataset from Huawei's real-world business services on HarmonyOS. Results demonstrate that SLOPE achieves an average parsing accuracy improvement of 61.2% and 5.9 times higher throughput over all baselines. Furthermore, SLOPE maintains a high parsing accuracy with an average drop of only 1.4% under diverse log inconsistencies, exhibiting state-of-the-art robustness.
Shuting Lai, Haiyu Huang, Pengfei Chen et al.· ACM Transactions on Software...· 0 citations
Agent systems rely on LLM APIs for every response, but these APIs can return server errors, truncated responses, or corrupted content that propagates through downstream agents and causes task failure. Evaluating robustness under these faults is crucial for reliable deployment. Existing fault injection methods are offline, require source code modification, or cannot modify specific response fields. A comprehensive evaluation also requires a systematic fault taxonomy because different fault types affect downstream agents differently. We propose AgentChaos, a chaos engineering framework for controlled, runtime, non-intrusive LLM API fault injection. Since all agent systems access LLMs through the same HTTP interface, we inject faults at this shared layer without modifying source code. We define crash, omission, and value faults on content and tool call fields, intercept and modify LLM API responses at runtime, and verify whether each fault is triggered to filter untriggered tasks and avoid underestimating fault impact. Evaluations across agent systems, benchmarks, and backbone LLMs under 65 fault configurations show that all systems degrade under fault injection, with pass@1 dropping by up to 50 percentage points. The ranking is consistent across models, suggesting that robustness depends on system implementation rather than model capability. Existing fault diagnosis methods achieve below 53% accuracy on fault type and below 56% on fault step, leaving room for improvement. We further reveal practical findings for agent system developers.
Gou Tan, Zhensu Sun, Jieke Shi et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
