Slowing Down Forgetting in Continual Learning: Evidence from Manufacturing

Abstract

Continual learning (CL) aims to incrementally update machine learning models from a stream of data without forgetting previously acquired knowledge. CL is highly relevant in many real-world applications such as manufacturing, where storing historical data for retraining is often infeasible due to volume, governance, or system constraints. Yet, a common challenge in continual learning (CL) is catastrophic forgetting, where the performance on old tasks drops after new, additional tasks are learned. In this paper, we propose a novel framework called ReCL to slow down forgetting in CL. Our framework exploits an implicit bias of gradient-based neural networks due to which these converge to margin maximization points. Such convergence points allow us to reconstruct old data from previous tasks, which we then combine with the current training data. Our framework is flexible and can be applied on top of existing, state-of-the-art CL methods. We first demonstrate the performance gain from our framework across a large series of experiments on three established public CL datasets (MNIST, CIFAR10, TinyImageNet), a public industrial dataset (SECOM), and across two different scenarios (class incremental and domain incremental learning). Then, we evaluate the performance of ReCL for predictive maintenance in a manufacturing environment at Bosch, a global engineering company, using an internal real-world time-series feature dataset captured from a high-volume precision-machining process. Lastly, we apply our framework to streaming machine data from real-world industrial data at Bosch. Across all our experiments, we find large performance gains through ReCL. To the best of our knowledge, our framework is the first to address catastrophic forgetting by leveraging models in CL as their own memory buffers.