RefSituational Awareness — AI Progress and the Possibility of a Slow Takeoff

Situational Awareness — AI Progress and the Possibility of a Slow Takeoff

Situational Awareness — AI Progress and the Possibility of a Slow Takeoff

Paul Christiano et al., situational-awareness.ai, June 2024.

Situational Awareness — AI Progress and the Possibility of a Slow Takeoff (website)

Full paper (PDF, 21 MB): https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf

A slow takeoff looks like this: AI systems are deployed and gradually improve, and at some point they become capable enough to pose a serious risk — but the transition from "deployable" to "dangerous" is gradual, giving people time to notice and respond. A fast takeoff (or "foom") would be a rapid, discontinuous jump that leaves little time to react.

The paper argues that forecasts of AI progress should take seriously the possibility that AGI-level systems could emerge from continued scaling + post-training improvements ( RLHF, automated evaluation, chain-of-thought, etc.) rather than needing a fundamental breakthrough. It models a scenario where AI systems become increasingly competent at autonomous replication, power-seeking, and strategic behaviour over a period of months to years — a "slow takeoff" that is still fast enough to be alarming.

Key themes:

  • Situational awareness — AI systems that understand their own position, their deployment context, and can act strategically in the world.
  • Post-training improvements as a major driver: RLHF, scaling, automated feedback, chain-of-thought, tool use — these compound and can push models from "not deployable" to "dangerous" without a sharp discontinuity.
  • Autonomous replication as a warning sign: a system that can copy itself, acquire resources, and persist — even in a limited way — is a threshold the paper takes seriously.
  • The paper is written in a forecasting / scenario-planning style (probabilistic over possible timelines), not a technical alignment proposal.

The most important question is not "will AGI happen?" but "what will the trajectory look like, and how much time will we have?"

The paper was published in June 2024, well before GPT-5-era results, and is part of a genre of early-2020s safety forecasting that takes slow-but-accelerating takeoff seriously. It predates the 2026 insider-warning wave (Coxon, Pachocki, the 1,384-employee statement) but shares the framing that "the pace is the story."

Links

Note: This is a 2024 paper. The insider-warning wave of Sep 2026 (Coxon resignation, Pachocki essay, 14+ endorsers, Gioia collection) happened after this was written and gives the "slow takeoff is still alarming" framing additional empirical weight.

Edit this page

Links to this page:

  1. 2026-09-13 Situational Awareness paper (Paul Christiano et al., 2024) — PDF cached + ref entry
  2. Speculations Concerning the First Ultraintelligent Machine