Cluster Indices for Real‐Time Monitoring

Abstract

As technology advances, the volume, variety, and velocity of data generation continue to grow, leading to the emergence of big data analytics, which aims to extract valuable insights from these extensive datasets. The increasing volume of data presents several opportunities and challenges. In the context of Statistical Process Monitoring (SPM), analyzing high‐dimensional data can lead to the “curse of dimensionality”, where the data becomes sparse, making it difficult to detect patterns and anomalies. Modeling complex variable relationships is also challenging. Implementing Multivariate SPM (MSPM) in real‐time settings is difficult due to the need for rapid computation and decision‐making, especially when dealing with large volumes of streaming data. A possible solution is to combine MSPM approaches with Machine Learning methods; however, challenges arise related to model interpretability, feature selection, and integration of results. In this paper, we propose a robust method based on a strategy from cluster analysis. The method is compared to multivariate control charts based on the Hotelling statistic and to Dunn's index. An extensive simulation study showed that the proposed method outperforms its competitors when data streams consist of correlated features.