Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation

Abstract

Łearning with Noisy Labels (LNL) is a challenge that arises in many scenarios where training data can contain noisy labels. While various methods, such as active learning for small language models (SLMs), have been proposed to re-annotate samples, they still require human efforts. The prevalent in-context learning (ICL) of large language models (LLMs) can perform text annotation tasks, but their efficiency hinges on the precise selection of clean versus noisy samples from noisy data. Meanwhile, this paper observes that existing sample selection suffers from selection and training bias on class imbalance noisy datasets, leading to decreased accuracy in sample selection. To bridge it, we pro-pose a debiased sample selection and an innovative collaborative learning framework DeCo based on active learning for LNL. During collaborative learning, we first conduct debiased sample selection by designing a robust expert model based on SLMs and introducing a dynamic class-wise threshold strategy, then feed selected clean samples to active annotator LLMs for re-annotating noisy samples us-ing ICL, with the re-annotation results reinforcing SLMs’ training for subsequent accurate selection. Ultimately, we employ distinct loss functions adept at managing subsets with varying degrees of label noise. Extensive experimental results on synthetic and real-world datasets demonstrate the effectiveness and superiority of our proposed method.