Scalable and Adaptive Log-based Anomaly Detection: A Synergistic Approach
TL;DR
SeaLog is presented, an accurate, lightweight, and adaptive log-based anomaly detection framework that synergistically integrates the efficiency of traditional data-driven methods with the zero/few-shot learning ability of large language models (LLMs).
Abstract
System logs are critical for software reliability. While many automated log-based anomaly detection methods exist, they often falter in large-scale cloud systems due to high resource consumption and poor adaptability to evolving logs. In this paper, we present SeaLog, an accurate, lightweight, and adaptive log-based anomaly detection framework that synergistically integrates the efficiency of traditional data-driven methods with the zero/few-shot learning ability of large language models (LLMs). SeaLog consists of a lightweight detection agent and a backbone analyzer. The detection agent utilizes an n-gram probabilistic tree (NPT) for efficient anomaly detection. The backbone analyzer employs an LLM enhanced by in-context learning (ICL) to provide accurate and adaptive predictions, especially for unseen log data. Both components continuously benefit from anomalies confirmed by on-site engineers. We evaluate SeaLog on three public datasets (BGL, Thunderbird, and HDFS) and an industrial dataset, with multiple LLM backbones, including open-source models, to assess its robustness to model choice. SeaLog achieves strong effectiveness across all datasets (F1 scores of 0.949–0.999), runs 2× to 5× faster, and consumes only 5% to 41% of the memory resources. Furthermore, we have successfully deployed SeaLog in Huawei Cloud, from which we share our insights and experiences during this deployment.