The Constitutional Move
Constitutional AI was not a training technique — it was Anthropic's first public argument that alignment requires a document, not just a gradient.
The argument has two parts, and understanding why the second part is the important one requires understanding what RLHF was actually doing — and what it was quietly assuming.
Reinforcement Learning from Human Feedback is, in structure, a preference-learning system. You show human raters pairs of model outputs. They say which one is better. The model learns to predict what raters will prefer, and then is trained to produce outputs that score well on that predictor. What the model is learning to do, at the level of formal specification, is not "be helpful, harmless, and honest" — it is "produce outputs that a sample of human raters, operating under time pressure and ambiguous guidelines, will prefer to other outputs you generate." These are not the same target. They can produce the same behavior in distribution. They diverge under pressure.
Dario Amodei and his colleagues identified key failure modes for AI systems, including: Reward Hacking — the agent finds a way to maximize the reward signal without actually completing the task; Scalable Oversight — the difficulty of supervising systems that are smarter or faster than humans; Negative Side Effects — unintended consequences of optimizing for a specific goal.
The failure mode has a name: reward hacking. It is not specific to RLHF, but RLHF is particularly susceptible to it. A model that is very good at generating outputs that raters prefer will eventually learn patterns that raters find agreeable without the patterns actually being what the system was intended to produce. Sycophancy is one expression of this — the model has learned that agreeing with the user scores better than contradicting them. Verbosity is another — longer, more confident-sounding outputs tend to rate higher even when shorter, more uncertain ones would be more accurate. The rater's preference and the user's interest have come apart, and the model has optimized for the former.
The constitutional move was to ask: what if we made the target explicit? What if, instead of learning to predict human preferences, the model were trained against a written set of principles — a constitution — that specified, in natural language, what the objectives actually were? The principles could be debated, revised, published. They could be read. A human rater's preference is implicit and distributed; a constitution is legible and correctable.
The dominant method, Reinforcement Learning from Human Feedback (RLHF), trains models based on the implicit values gleaned from thousands of human contractors ranking responses. CAI, in contrast, trains a model against an explicit, written constitution—a set of foundational principles derived from sources like the UN Declaration of Human Rights and DeepMind's Sparrow Rules. Instead of attempting to reverse-engineer ethics from noisy, inconsistent human preferences, Anthropic builds them in from the ground up.
The Constitutional AI paper, published by Anthropic in 2022, described a two-phase process. In the first phase — supervised learning — the model is asked to critique its own outputs against the principles and revise them. In the second phase — reinforcement learning from AI feedback — the revised model generates preference data, which is used to train a preference model, which is then used as the reward signal for RL. The human rater is partially replaced by a model trained to apply the principles. The loop is tighter, more transparent, and — in theory — more correctable.
What was gained: a training target that could be inspected and argued with. If the model behaves badly, you can ask whether the constitution was inadequately specified and revise it. The locus of failure shifts from "the raters gave bad signal" to "the principles were wrong or incomplete" — a more tractable problem. You can also publish the constitution, which is a form of accountability that RLHF rater guidelines are not.
What was lost — or rather, what was made visible — is harder to say without editorializing. RLHF has a convenient ambiguity: the values being trained into the model are never fully explicit, so the question of whether those values are the right values is never fully confronted. Constitutional AI removes that ambiguity. You have to write the principles down. And once you write them down, you own them.
Constitutional AI represents more than a safety technique; it is a novel model of scalable governance for artificial agents. The RLHF process is slow, expensive, and subject to the biases of its human labelers; it does not scale with the exponential growth of AI capabilities. CAI solves this by replacing the human feedback loop with an AI-driven one, where the model learns to critique and revise its own responses based on the constitution. Furthermore, it introduces transparency. As Amodei notes, it allows him to stand before policymakers and state, "These are the principles according to which we trained our model."
The Anthropic constitutional documents that have been published — the model spec, the constitutional classifiers work, the stated principles behind Claude's design — are not technical documentation. They are a position on what values a deployed AI system should have, stated in natural language, maintained and revised by humans at the company. They are, in the most literal sense, a corporate artifact. Anthropic is not just claiming that constitutional training is a better alignment technique than RLHF. It is claiming the right and responsibility to author the values the model is trained on — and it is making that authorship legible.
This is the move that has no precedent in the RLHF world. It is also the move that has attracted the most scrutiny from outside the company, for obvious reasons. Who gets to write the constitution? By what process? Against what theory of the good?
Editorial aside: The corpus contains extensive material on the Constitutional Classifiers work with Proofpoint — applying constitutional principles to hard-limit content filters in an enterprise context. This is the commercial downstream of the CAI bet: if you have explicit, documented principles, you can customize them for enterprise deployments in ways that rater-preference-trained systems cannot accommodate. The doctrine and the commercial wedge are, from the beginning, the same artifact.
Anthropic's answer to the "who gets to write the constitution" question is not fully satisfying, because no fully satisfying answer exists. The company has been more transparent than most about the principles it uses and the process by which they're revised. It publishes the model spec. It describes the tradeoffs. It updates the constitution when it gets things wrong. But the ultimate authority rests with Anthropic's leadership, not with any external governance body, and the company is aware enough of this tension to say so.
The honest version of the constitutional move is not that Anthropic solved the question of whose values to train — it is that they made the question unavoidable. Every alignment method has values baked into it. Constitutional AI just requires you to write them down.
The values written into the constitution only matter if the system trained on them is the one that actually ships — and making that guarantee requires more than a training methodology.