Beyond Pseudo-Labels: Dual-Level Knowledge Distillation for Enhanced Deep Clustering

Abstract

This study introduces a novel clustering approach, namely Teacher–Student-based Deep Clustering (TSDC), that relies on intra- and inter-distillation-based feature representations. In fact, TSDC distils knowledge from (i) high- to low-response channels, forcing the latter to mimic the former, and (ii) deeper to shallower layers, prompting the transfer of semantic information to enhance representational consistency. Unlike CNN-based deep clustering that relies on pseudo-labels to improve representations, TSDC introduces a newly formulated objective function to simultaneously minimize losses from clustering and intra- and inter-distillation. The proposed approach was rigorously investigated using benchmark datasets and relevant performance measures. In particular, a linear top classifier protocol was adopted to assess TSDC performance. Notably, TSDC outperformed existing deep clustering frameworks, yielding improved classification accuracy. The empirical findings highlight the efficacy of combining intra- and inter-distillation to enrich feature representations. Notably, the most prominent improvement was observed on CIFAR-100, where classification accuracy rose from 54.77 ± 0.07 to 57.58 ± 1.85.