Events, programs, and opportunities from MAIA. Join mailing list

(Mis)Alignment Through Reinforcement Learning

Fall 2026Week 2

← All weeks

This week, we examine the interaction between alignment and reinforcement learning. First, we will learn about reinforcement learning from human feedback (RLHF), one of the earliest and most important alignment techniques. We will study it in technical depth, including PPO, the reinforcement learning algorithm it commonly uses. Next, we will read a paper showing that LLMs tend to echo users' political views, a form of sycophancy that is partly caused by RLHF. We will conclude with two papers on how reinforcement can go wrong. The first shows how reward hacking can teach models harmful behavior, and the second shows how reinforcement learning can produce models that chase rewards while ignoring what their developers intended. The goal of this week is to understand the ways in which reinforcement learning can be both a solution and a risk for alignment.

Week 2 readings

Aligning language models to follow instructions

Ryan Lowe and Jan Leike, OpenAI

January 27, 2022

Training a Misaligned Reward Seeker

Richard Qi, Benjamin Wright, Monte MacDiarmid, and Evan Hubinger, Anthropic

August 2026

Measuring Reward-Seeking by Instilling Contrastive Beliefs

Axel Højmark, Jérémy Scheurer, Jenny Nitishinskaya, Felix Hofstätte, Jasoxfn Wolfe, Theodore Ehrenborg, Bronson Schoen, and Alexander Meinke, Apollo Research and OpenAI

July 21, 2026

Additional readings