Surface Matching in Skill Recommenders: Incomplete Evaluation Hides Simplicity Bias
TL;DR
Two training-time mitigations, label augmentation and shortcut-targeted hard negatives, which reduce judged nonsense while preserving ranking performance are studied.
Abstract
Skill-extraction models are typically evaluated with ranking metrics scored against human-annotated positives. At the scale of ESCO (13,939 skills), annotation is inherently incomplete, and unannotated labels are treated as uniformly wrong irrespective of semantic proximity. Training sentences, dominated by LLM-generated synthetic data, mention many skills (almost) verbatim, so any model trained on them is prone to simplicity bias: exploiting surface-level lexical, or partly semantic, overlap between the input text and skill label as a shortcut. We demonstrate this on the popular contrastive bi-encoder setup. Annotated positives and the shortcut often coincide, so ranking metrics remain high even when other top predictions are semantically nonsensical. We use an LLM-as-a-judge, validated against human labels, to flag such nonsense at scale, and isolate the shortcut with a controlled diagnostic we construct automatically from the taxonomy, in which every distractor shares a content token with the gold skill yet comes from a clearly different domain. We then study two training-time mitigations, label augmentation and shortcut-targeted hard negatives, which reduce judged nonsense while preserving ranking performance. Our findings show how incomplete annotations silently degrade skill recommender quality, and give practical diagnostics and training-time levers for this setting.