This thesis addresses two critical dimensions of Trustworthy AI and Efficient Multimodal Representation Learning: security through analyzing, detecting, and designing backdoor attacks in NLP and VLMs, and efficiency through advanced multimodal representation methods tailored for clinical and medical imaging applications.
Abstract
Recent advances in deep learning have significantly enhanced the capabilities of Natural Language Processing (NLP) and Vision-Language Models (VLMs). However, these advancements come with increased vulnerabilities, notably through backdoor attacks that pose severe security threats. This thesis addresses two critical dimensions of Trustworthy AI and Efficient Multimodal Representation Learning: (1) security through analyzing, detecting, and designing backdoor attacks in NLP and VLMs, and (2) efficiency through advanced multimodal representation methods tailored for clinical and medical imaging applications.
The recent progress in the vision-language model (VLM) research made it one of the key aspects of artificial intelligence due to joint vision and language processing capabilities required for such tasks as image retrieval, visual question answering, autonomous systems and medical image analysis. However, despite the ex...
Kumari Anjali, Karnatakam Veda Sahithi, Sakinala Jyotsna· 2026 4th International Confe...· 0 citations
This paper proposes a novel LVLM attack method, called BadPhase with further backdoor designs, to implant adversarial phase as triggers into any image inputs via data poisoning so as to control the LVLMs’ predictions and finds that LVLMs are sensitive to the phase-aware image structure.
Dai-Zong Liu, Junhao Dong, Xiang Fang et al.· Proceedings of the Thirty-Fi...· 0 citations
This paper identifies three previously overlooked issues caused by inappropriate cross-modal interactions and excessive operations in the Simple Vision-Language Attack (SimVLA) pipeline, and proposes the SimVLA, which observably improves transferability and efficiency.
Yuchen Ren, Zhengyu Zhao, Chenhao Lin et al.· IEEE Transactions on Informa...· 0 citations
To efficiently generate adversarial examples, a gradient-based attack method is proposed that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture, which significantly reduces the computational cost and resource requirements of the attack while maintaining...
Ilan Zini, B. Addad, Katarzyna Kapusta· 1 citation
A new perspective of information theory is introduced to investigate LVLMs’ transferable characteristics by exploring the relative dependence between outputs of the LVLM model and input adversarial samples and formulate the complicated calculation of information gain as an estimation problem and incorporate such inform...
Xiaowen Cai, Daizong Liu, Xiaoye Qu et al.· Neural Information Processin...· 8 citations
This survey provides a comprehensive overview of existing research at the intersection of LLMs and FL, and outlines promising future research directions to advance the scalability, efficiency, and ethical deployment of LLMs in federated settings, paving the way for more trustworthy and privacy-preserving natural langua...
Fatiha Ait Baali, Chaima Lhasnaoui, Addi Ait-Mlouk et al.· Machine-mediated learning· 0 citations
