Zi-Shuo Zheng

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed,...