Events, programs, and opportunities from MAIA. Join mailing list
Character and Inner Alignment
This week, we begin by discussing alignment through the perspective of instilling “good character” in models. We start with Emergent Misalignment, which demonstrates that models trained on harmful code completions generalize to broadly misaligned behavior on unrelated tasks. To help explain this surprising result, we turn to Anthropic’s Persona Selection Model, which proposes a mental model of LLMs as actors capable of simulating a vast repertoire of characters. From this perspective, a relatively small amount of flawed training can push models away from the Assistant persona and towards problematic personas. We then examine Alignment Faking, which shows that models may strategically comply with its training objective to prevent its existing preferences from being changed. We end with mesa-optimization and inner alignment, examining how a model can learn to pursue an objective that produces the desired training behavior but undesirable deployment behavior. The goal of this week is to conclude our discussion of (mis)alignment with a broad picture of how training shapes an LLM’s character and how this process can produce behavior we didn’t intend.
Week 4 readings
Additional readings
- Toward understanding and preventing misalignment generalization
- Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
- Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Frontier Models are Capable of In-context Scheming
- Risks from Learned Optimization in Advanced Machine Learning Systems
- The third wave of American philanthropy
- Teaching Claude Why
- Model Spec Midtraining: Improving How Alignment Training Generalizes
- Reinforcement Learning Towards Broadly and Persistently Beneficial Models
- The “guerilla warrior” who taught OpenAI to fight
- Why Do Some Language Models Fake Alignment While Others Don't?