Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs
Long-context language models often neglect information in the middle of the input, a phenomenon known as"lost-in-the-middle". Part of this gap may stem from the long-range decay of rotary position embedding (RoPE), which weakens the positional signal between distant tokens. Existing input compression deletes low-import...