Parameter-efficient cross-modal prompt tuning for few-shot ancient mural classification
TL;DR
PE-CPT adapts a pre-trained vision-language model by learning a small set of prompt vectors without updating its original parameters, and incorporates LoRA, a parameter-efficient adaptation method applied to attention and feed-forward layers to improve adaptation.
Abstract
Ancient murals are invaluable cultural heritage, and their automatic classification is crucial for digital preservation, stylistic analysis, and heritage management. Existing approaches require large, high-quality annotated datasets, but such expert annotations are costly and often unavailable. To address this, we formulate a few-shot ancient mural classification problem and propose a Parameter-Efficient Cross-modal Prompt Tuning (PE-CPT) framework. Specifically, PE-CPT adapts a pre-trained vision-language model by learning a small set of prompt vectors without updating its original parameters. It performs cross-modal prompt tuning between image and text encoders, generating image prompts from text prompts to maintain semantic consistency. To further improve adaptation, we incorporate LoRA, a parameter-efficient adaptation method applied to attention and feed-forward layers. Experiments in a 16-shot setting demonstrate that PE-CPT outperforms state-of-the-art baselines, achieving improvements of 6.79%, 6.59%, and 7.07% in accuracy, precision, and F1 score. These results demonstrate its effectiveness in cultural heritage analysis under data scarcity.