Beyond Gut Feel: An Empirical Foundation for AI-Assisted Observability Design

Abstract

Modern software systems rely on observability infrastructure (metrics, logs, and traces) to detect failures and maintain reliability. In practice, key observability decisions are made through untested defaults: static alerting thresholds, keyword-based log-level selection, and monitoring feature sets chosen by convention rather than evaluation. Despite growing interest in AIOps, no prior study has directly measured the gap between these default rule-based practices and machine learning across both anomaly detection and logging decisions. We present a two-part empirical study quantifying this gap. In Part A, we evaluate static-threshold baselines (𝜇 ± 3𝜎) against per-KPI machine learning models on the AIOps 2018 benchmark, which comprises 2.67 million labelled data points. ML models outperformed the static threshold on 91% of evaluable KPIs, though absolute performance varied widely (mean F1 = 0.41, median = 0.116), and an ablation study showed that 53% of monitoring features could be removed without degrading detection. SHAP analysis confirmed that no universal feature ranking exists across KPIs. In Part B, we mine 15,702 log statements from 15 open-source Node.js/TypeScript repositories and train classifiers to predict developer-chosen log levels from code context. The model achieves cross-project macro F1 of 0.92 when surrounding code (including existing log statements) is available, compared to 0.38 for a keyword heuristic. However, a level-name-stripped ablation reveals this drops to 0.52 without neighbouring log-level tokens, showing the dominant signal is inter-statement level clustering rather than deeper code-structural patterns. This clustering generalises across 15 independent codebases and 3 unseen test repositories. Our results demonstrate that default observability practices leave substantial performance on the table, while identifying boundary conditions where simple rules remain competitive.