Interaction of Large Language Models and Web Information Retrieval & Ranking: Frameworks, Challenges and Future Prospects
TL;DR
This paper aims to study the synergy of LLMs and web Information Retrieval (IR) systems, consolidate existing research progress, and provide an overview of their interactive mechanisms to overcome mutual limitations.
Abstract
: As the global number of websites has surged to new high, leaving traditional search technologies increasingly inadequate in addressing complex query requirements together with fulfilling demands for accurate, contextually relevant and real-time responses. Recently Large Language Models (LLMs) have made remarkable progress in terms of natural language processing and content generation. But they suffer from limited real-time knowledge and hallucinations. This paper aims to study the synergy of LLMs and web Information Retrieval (IR) systems, consolidate existing research progress, and provide an overview of their interactive mechanisms to overcome mutual limitations. The study focuses on two paradigms: 1) Search4LLM, where search engines support LLMs throughout their life cycle by providing diverse data and refining relevance; 2) LLM4Search, where LLMs enhance search modules via four key capabilities. This paper further elaborates three practical frameworks: 1) InteR, this is a design to conduct information refinement with the interaction of Retrieval Models (RMs) and LLMs ; 2) Pyramid-ERNIE, an architecture designed to solve challenges of applying Pre-trained Language Models(PLMs) to large-scale web search; 3) SAT-Ranker, a PLM-based framework designed to model multi-dimensional user satisfaction to replace traditional relevance-focused ranking. This paper also analyzes critical challenges and outlines future directions respectively.