AI Alignment Curriculum

As of early February, 2023, we updated the curriculum to match the one being used for the ongoing cohort, which involved significant changes to some sections and readings. If you were working through the previous version of the curriculum and want to finish it, see this Google doc.

Key points

The AGI Safety Fundamentals alignment curriculum provides a high-level understanding of the AI alignment problem and some of the key research directions which aim to solve it.

The curriculum was compiled, and is maintained, by Richard Ngo; this version was last updated February 2022.

The curriculum is intended to be accessible for people with a wide range of backgrounds; those who are already familiar with some readings can choose substitutes from the Further Readings section for that week. The curriculum doesn't aim to teach practical programming or machine learning skills; those who primarily want to upskill for alignment work should instead take the courses listed on our resources page.

Course details

The course consist of 7 weeks of readings and discussion sessions (plus one optional week introducing the fundamentals of machine learning), and a final project. No background machine learning knowledge is required, but participants will be expected to have some fluency in basic statistics and mathematical notation.

The main focus each week will be on the core readings and one exercise of your choice out of the exercises listed, for which you should allocate around 2 hours preparation time. Most people find some concepts from the readings confusing, but that’s totally fine - resolving those uncertainties is what the discussion sessions are for.

Broadly speaking, the first half of the course explores the motivations and arguments underpinning the field of AGI safety, while the second half focuses on proposals for technical solutions. After week 7, participants will have several weeks to work on projects of their choice, to present at the final session.

Approximate times taken to read each piece in depth are listed next to them. Note that in some cases only a small section of the linked reading is assigned. In several cases, blog posts about machine learning papers are listed instead of the papers themselves; you’re only expected to read the blog posts, but for those with strong ML backgrounds reading the paper versions might be worthwhile.

If you’ve already read some of the core readings, or want to learn more about the topic, then the further readings are recommended; see the notes for descriptions of them. However, none of them are compulsory. Also, you don’t need to think about the discussion prompts in advance - they’re just for reference during the discussion session.