Events, programs, and opportunities from MAIA. Join mailing list
(Mis)Alignment Through Reinforcement Learning
This week, we examine the interaction between alignment and reinforcement learning. First, we will learn about reinforcement learning from human feedback (RLHF), one of the earliest and most important alignment techniques. We will study it in technical depth, including PPO, the reinforcement learning algorithm it commonly uses. Next, we will read a paper showing that LLMs tend to echo users' political views, a form of sycophancy that is partly caused by RLHF. We will conclude with two papers on how reinforcement can go wrong. The first shows how reward hacking can teach models harmful behavior, and the second shows how reinforcement learning can produce models that chase rewards while ignoring what their developers intended. The goal of this week is to understand the ways in which reinforcement learning can be both a solution and a risk for alignment.
Week 2 readings
Additional readings
- Recent Frontier Models Are Reward Hacking
- Specification gaming: the flip side of AI ingenuity
- Learning from human preferences
- Policy Gradients
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Language Models Learn to Mislead Humans via RLHF
- Scaling Laws for Reward Model Overoptimization
- Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?
- Risk from fitness-seeking AIs: mechanisms and mitigations
- RL creates split personas
- How training-gamers might function (and win)
- The behavioral selection model for predicting AI motivations
- Physics of RL: Toy scaling laws for the emergence of reward-seeking
- Investigating reward tampering in language models
- Modifying LLM Beliefs with Synthetic Document Finetuning