Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering
Visual contrastive decoding corrects answers using the prediction difference between the original image and a reference condition, yet the same contrast also turns supported abstentions into incorrect answers. We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the...
Hao-Nan Zhang, Ting Jin, Wen-Zhong Tang et al.
· 0 citations