Self-play red-teaming improves language-model safety by pitting attacker and defender roles against each other in a zero-sum game. However, real adversaries increasingly use mosaic attacks: multi-turn sequences whose individual fragments are innocuous in isolation yet assemble into a harmful payload. We develop a theor...
Emanuele La Malfa, Saar Cohen, G. La Malfa et al.· 0 citations
Personal advice, including relationship advice, now ranks among the most common uses of generative AI. But language models (LMs) exhibit sycophancy: they affirm users much more often than humans do, which can make people overconfident and less willing to repair their relationships after a conflict. Prior work on mitiga...
Stephane Hatgis-Kessell, Myra Cheng, Xiao-Xuan Hou et al.· 0 citations
Large language models (LLMs) can now improve themselves by revising the instructions they follow, and LLM agents are increasingly orchestrated to work together on complex problems. However, self-improvement methods typically optimize one system at a time, and multi-agent frameworks often have every model work toward a...
Kunal Jha, Max Kleiman-Weiner, Natasha Jaques· 0 citations
Autopoietic Game Theory is introduced, a computational model where social interactions, replication mechanisms, and their associated computational costs are endogenous and simultaneously co-evolving and can incorporate exogenous pressures.
Kunal Jha, F. Cicala, B. A. Y. Arcas et al.· 0 citations
SPADE (Self-Play in Adaptive Synthetic Executable Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that le...
Bo Liu, Simon Yu, Yiding Jiang et al.· 7 citations
This work deconstructs the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface, and uses the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shape...
D. Clay, Saket Gollapudi, Sankar V Harilal et al.· 1 citation
We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking compared to a reinforcement learning from AI feedback (RLAIF) baseline. Reward hacking is a central obstacle in RLAIF: as training progresses, t...
Zachary Kenton, Lili Janzer, Rory Greig et al.· 1 citation
SPADE (Self-Play in Adaptive Synthetic Executable Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that le...
Bo Liu, Simon Yu, Yiding Jiang et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.