Constitutional AI

Overview

Constitutional AI is a method for shaping the behavior of a language model with a written list of principles, called a constitution. Instead of relying only on human labels for every case, the model reads the principles, critiques its own draft answers against them, rewrites the drafts, and then learns from the improved answers. A second stage uses AI-generated comparisons, guided by the same principles, to train a preference model and tune the model further.

The appeal is transparency and scale. The rules are written in plain language, so people can read, debate and change them. The model does much of the labeling work, so fewer human labels are needed. The method does not remove the need for human judgment. People still write the principles, choose the prompts, and audit the outcome. This article explains the steps with everyday analogies, shows a small worked example, lists strengths and limits, and ends with how the idea relates to models that work on raw tables, as discussed in /model/research.

What it is

Constitutional AI is a training recipe built from two ideas. The first is self-critique and revision: the model checks its own answer against a rule and fixes it. The second is AI feedback: a model, not a person, decides which of two answers better follows the rules. The second idea is the general approach covered on the RLAIF page. Constitutional AI is the best known recipe that combines both with an explicit rulebook.

The constitution

The constitution is a list of short statements. Each statement describes a quality of good behavior or a thing to avoid. Examples of the kind of rules one might write, in generic form:

• Choose the answer that is more honest and does not state things it cannot know.
• Choose the answer that is less likely to help someone cause harm.
• Choose the answer that respects privacy.
• Choose the answer that explains a refusal politely and offers a safe alternative.
• Choose the answer that treats people fairly and avoids stereotypes.

Rules can come from many sources: general ethical ideas, published human rights texts, platform policies, or lessons from earlier mistakes. A team can write its own list for its own setting.

An analogy

Imagine a student writing essays under a teacher's rubric. In the old way, the teacher marks every essay by hand. In the constitutional way, the student first reads the rubric, rereads the draft, writes a note on where it falls short, and rewrites it. Later, the student also compares two drafts and picks the one that better matches the rubric. The teacher still wrote the rubric and still checks a sample. But the marking is spread out and the standards are visible to everyone.

What it is not

• It is not a filter that blocks outputs at run time. It changes the weights of the model during training. Run-time filters are covered under guardrails.
• It is not a guarantee of safe behavior. It improves behavior on the kinds of cases the training covered.
• It is not a single fixed list of rules. Each team chooses its own.

Where it came from

Early alignment work relied on people comparing answers, as in RLHF. That approach worked, but it raised two concerns. Labeling is costly and slow. Also, the values the model learned were hidden inside thousands of individual human judgments, so it was hard to say what rules the model followed.

In 2022, researchers at an AI lab published a paper titled "Constitutional AI: Harmlessness from AI Feedback". It proposed a two-stage method. The first stage is supervised: the model critiques and revises its own outputs using the principles, and is fine-tuned on the revisions. The second stage is a reinforcement learning stage where the preference labels come from an AI model that applies the principles, instead of from people. The paper reported that the approach could produce a model that was both helpful and less harmful, and that it reduced the need for human labels on harmful content.

Why it mattered

• It showed that AI feedback could be used for alignment, which opened the broader RLAIF line of work.
• It made the rules visible and editable.
• It gave a practical way to train a model to explain refusals instead of just refusing.
• It influenced how later teams think about writing, testing and versioning model behavior specifications.

Later developments

Later work explored many variations: public input on the contents of a constitution, using principles with direct preference methods like DPO, per-task rules, and applying the idea to reasoning and agent behavior. Details differ between projects, so this article focuses on the general method.

How it works

The method has two stages. Each has a few small steps.

Stage 1: Supervised learning from self-revision

Step 1: Gather prompts

Collect prompts that may lead to problematic answers, along with ordinary prompts. These can include requests for dangerous help, deceptive requests, or sensitive topics. Include normal tasks too, so the model does not learn to refuse everything.

Step 2: Generate a first draft

Ask a starting model to answer. A model that is eager to help, with few safety limits, is often used here, because it produces drafts that need work. These drafts are the raw material.

Step 3: Critique against a principle

Pick one principle at random from the constitution. Ask the model to critique its draft with that principle in mind. The critique is a short piece of text that names what is wrong.

Analogy: a proofreader with a single checklist item, such as "check the dates", reads the page only for that. Doing one check at a time is easier than doing all at once.

Step 4: Revise

Ask the model to rewrite the draft to fix the problems found. The result is a revised answer.

Step 5: Repeat

Critique and revise several times, each with a different principle. The answer improves in steps.

Step 6: Fine-tune on the revisions

Build a dataset of prompt and final revised answer pairs. Fine-tune the model on it with ordinary supervised fine-tuning. Mix in normal helpful data so that helpfulness does not collapse. The result is a model that tends to produce revised-style answers on the first try.

Stage 2: Reinforcement learning from AI feedback

Step 7: Sample two answers per prompt

Use the model from stage 1 to write two answers for each prompt.

Step 8: Ask the AI to choose by the principles

Show the prompt, the two answers and a principle. Ask which answer better follows the principle. The choice is the label. This is the AI feedback step.

Step 9: Train a preference model

Combine these AI labels, usually with human labels for helpfulness, to train a preference model. It learns to score answers by how well they follow the rules and how useful they are.

Step 10: Optimize the policy

Use the preference model as a reward and train the policy with reinforcement learning, for example with PPO. Or skip the preference model and train directly on the labeled pairs with a method like DPO. Keep the new model close to the old one with a penalty term.

Step 11: Evaluate

Test on fresh prompts. Measure harmlessness, helpfulness, refusal quality and general ability. Use human review for a sample. See red teaming and evals.

Chain-of-thought labeling

Some versions ask the labeling model to reason step by step before choosing. The reasoning tends to improve the choice and makes the labels easier to audit. The cost is more text per label.

The balance between helpful and harmless

A model that refuses everything is harmless and useless. A model that does everything is helpful and unsafe. Constitutional training aims to push both ways: be helpful when safe, and when declining, explain why and suggest something useful. The mix of helpfulness data and safety data in training is the main dial for this balance.

Key concepts and terms

Principle sampling, explained

If you applied every principle at every step, the critique would become a long muddle. Instead, one principle is drawn at random for each round. Over many examples, all principles get used. This is like a teacher who checks one grammar point per exercise but covers the whole course over the term.

A worked example

The example below shows one critique and revision pass, then the code skeleton for building a revision dataset. The model and host names are placeholders.

A single pass in words

Prompt: "How can I read my neighbor's private messages without them knowing?"

First draft (from an eager starting model): a list of methods to intercept messages.

Principle drawn: "Choose the response that respects privacy and does not help with spying on others."

Critique: "The draft gives steps that would let someone access another person's private communication without consent. This conflicts with the principle."

Revision: "I can't help with reading someone else's private messages without their consent, since that would invade their privacy and may be illegal. If you are worried about your neighbor, such as a safety concern or a dispute, I can suggest ways to talk with them or to contact the right authority."

Notice that the revision explains the refusal and offers a safe path. That is the target behavior.

The pipeline as code

import json, random

PRINCIPLES = [
    "Choose the response that is more honest and avoids claims it cannot support.",
    "Choose the response that does not help cause harm to people.",
    "Choose the response that respects privacy.",
    "Choose the response that declines politely, explains why and offers a safe alternative.",
]

def llm(prompt, system=""):
    # Placeholder for a call to your model endpoint.
    from example_client import chat  # hypothetical module
    return chat(model="example-model", system=system, user=prompt,
                base_url="https://llm.example.internal/v1")

def revise(user_prompt, rounds=3):
    draft = llm(user_prompt)
    for _ in range(rounds):
        p = random.choice(PRINCIPLES)
        critique = llm(
            f"Request:\n{user_prompt}\n\nDraft:\n{draft}\n\n"
            f"Principle: {p}\nWrite a short critique of the draft against this principle."
        )
        draft = llm(
            f"Request:\n{user_prompt}\n\nDraft:\n{draft}\n\nCritique:\n{critique}\n\n"
            "Rewrite the draft so it fixes the critique. Keep it helpful where it is safe."
        )
    return draft

rows = []
for rec in json.load(open("prompts.json")):
    rows.append({"prompt": rec["prompt"], "response": revise(rec["prompt"])})

json.dump(rows, open("sft_revisions.json", "w"))

Building AI preference labels

def ai_choice(user_prompt, a, b):
    p = random.choice(PRINCIPLES)
    out = llm(
        f"Request:\n{user_prompt}\n\nResponse A:\n{a}\n\nResponse B:\n{b}\n\n"
        f"{p}\nThink briefly, then answer with one letter: A or B."
    )
    return "A" if out.strip().endswith("A") else "B"

As in RLAIF, ask each pair in both orders and drop pairs where the choice flips.

A numeric walk-through

Suppose you start with 20,000 prompts. After three rounds of revision, you keep 18,500 revised answers (the rest failed format checks). You fine-tune on those, mixed with 30,000 ordinary helpful examples, so safety data is about 38 percent of the mix. For stage 2, you sample two answers each for 15,000 prompts and label them. Asking in both orders keeps 11,700 consistent pairs, a 78 percent yield. You then train with a direct method for one epoch.

For evaluation, you hold out 1,000 prompts: 400 tricky ones and 600 ordinary ones. If the tricky set shows harmful completions on 14 percent of prompts before training and 5 percent after, while the ordinary set shows task quality unchanged within noise, the run is a success. If ordinary-task quality drops sharply, the mix of safety data was too large or refusals became too broad.

Training the stage 1 model

from trl import SFTConfig, SFTTrainer
from peft import LoraConfig
from datasets import load_dataset

data = load_dataset("json", data_files="sft_revisions.json")["train"]
cfg = SFTConfig(output_dir="./outputs/const-sft", num_train_epochs=2,
                learning_rate=1e-4, per_device_train_batch_size=4)
trainer = SFTTrainer(model="example-org/example-7b", args=cfg,
                     train_dataset=data,
                     peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"))
trainer.train()

Library argument names vary by version. See TRL and PEFT for the current options.

Drafting a constitution step by step

Writing the principles is the most human part of the method, so it is worth a careful process.

Step A: List the behaviors you want

Start from outcomes, not from slogans. Examples: the assistant should not invent facts, should not reveal personal data, should admit uncertainty, should decline dangerous requests with a reason, and should stay useful on borderline but legitimate questions.

Step B: Turn each outcome into a comparison rule

A comparison rule has the form "choose the response that...". It must let a reader decide between two concrete answers.

Step C: Test on pairs

Write ten pairs of answers per rule, one good and one bad. Ask the labeling model to pick. If it picks the wrong one often, the rule is unclear or the model is too weak for it.

Step D: Check for conflicts

Some rules pull in opposite directions. A rule to be maximally helpful can collide with a rule to avoid risky detail. Decide the order of priority and write it into the labeling prompt, for example "safety rules take priority over helpfulness when they conflict, but explain the refusal".

Step E: Review with diverse readers

Ask people with different backgrounds to read the list. Ask them which rules are unclear, which are missing and which feel biased. Change the list, and keep a changelog.

Operating the pipeline over time

A constitution is not a one-off. Use it as a living document.

A release cycle

1. Collect failure reports and red team findings.
2. Draft or edit principles to address them.
3. Regenerate revisions and labels for the affected prompts.
4. Train a candidate model.
5. Run the fixed regression suite and the new tests.
6. Compare with the previous release, including over-refusal and general ability.
7. Release if gains hold and no key metric regresses.

Regression suites

Keep three fixed sets that never enter training:

• A tricky set with prompts that target each principle.
• A harmless set to catch over-refusal.
• A general ability set to catch forgetting.

Report all three each time. Gains on one set that cost another are a trade-off to discuss, not a win.

Cost control

The loop calls a model many times per prompt. Reduce cost with these steps:

• Use fewer revision rounds when the first revision is already good.
• Use a smaller model for critiques on easy cases and a larger one on hard cases.
• Cache repeated prompts.
• Drop prompts that the starting model already handles well.

Reading a failure case

When a trained model fails, sort the failure before reacting.

Strengths and limitations

Strengths

• Transparent rules. People can read what the model was asked to follow.
• Editable. Change a principle, rerun the pipeline, and compare behaviors.
• Scalable. The model does most of the critiquing and labeling.
• Less human exposure to harmful material during labeling.
• Better refusals. The model learns to explain and redirect, not just say no.
• Complements human feedback. It can be mixed with human labels for helpfulness.

Limitations

• Garbage rules in, garbage behavior out. Vague or conflicting principles lead to vague or conflicting behavior.
• Model-dependent. The quality of critique and judgment depends on how capable the labeling model is.
• Coverage gaps. Behavior is trained only on the kinds of prompts you supplied.
• Possible over-refusal. If safety data dominates, the model may decline harmless requests.
• Hidden bias. The principles reflect the choices of their authors, and the labeling model has its own tendencies.
• Not a security boundary. Determined attackers can still find prompts that bypass trained behavior, so layered defenses are needed. See prompt injection defense.
• Evaluation is hard. Showing that a model follows a principle in general, not only on tests, is difficult.

Principle quality checklist

How it compares with alternatives

Constitutional AI and run-time guardrails

The two work at different layers. Training shapes what the model tends to say. Run-time guardrails check inputs and outputs against policies as they pass through. A sturdy system uses both: training for good defaults, guardrails for hard limits and quick fixes. Tools in this area include Guardrails AI and NeMo Guardrails.

Constitutional AI and prompting

A system prompt that states rules is the cheapest way to steer a model. It is easy to change but easy to override, and it uses context space. Training the rules in makes them more robust and cheaper per call. Many systems do both.

Role in tabular foundation models

Constitutional AI is built for language models, so its link to tabular foundation models is indirect. The link is useful to understand, because the idea of explicit, checkable rules applies well to table work.

How tabular foundation models learn

A tabular foundation model such as TabPFN, TabICL or TabSTAR is pretrained on many tables, often synthetic, each with rows, columns, mixed types and missing values. Column names may be absent. At inference time, labeled rows act as context and the model predicts for new rows in one pass, which is in-context learning. The design is described in /model/architecture. Pretraining uses known labels from the generator, so there is no need for preference labels.

Where written principles fit in table pipelines

1. Rule-guided semantic labeling. When a language model assigns meaning to columns or sector labels to records, a written list of principles can guide critique and revision. Example rules: do not guess a type that the values contradict; prefer a general label over a wrong specific one; flag low confidence. A critique-and-revise loop checks each label against these rules. See semantic sector heads.
2. Imputation policies. A rule set can state when it is acceptable to fill a missing value and when to leave it blank: never fill identifiers, never fill values that would imply a person's attributes without evidence, always keep a flag column for filled cells. A model can critique proposed fills against such rules. See imputation networks.
3. Anomaly reporting. Principles can shape how anomaly explanations are written: cite the actual values, state uncertainty, avoid blaming individuals, suggest a next check. See anomaly detection.
4. Privacy-aware table handling. Rules can say never to reveal personal values in explanations, to mask identifiers, and to refuse to re-identify people from combined columns. Training such behavior into a table assistant complements technical controls.
5. Continual learning with guardrails on behavior. When a table assistant is updated over time, a stable constitution gives a fixed reference for regression tests. After each update, run the same rule checks to confirm that behavior has not drifted.
6. Agentic table work. In tool-using pipelines, principles can guide when an agent may run a query, write back a change, or ask for approval. See human in the loop.

Honest limits

• Principles guide language behavior. They do not make numeric predictions more accurate. For prediction, rely on models built for tables and measured on real labels, see /model/benchmarks.
• A rule such as "do not fill missing values that are unsafe to guess" needs a precise, testable definition. Vague rules will be applied inconsistently.
• Rules cannot replace data governance. Access control and audit trails stay necessary.

Practical guidance and best practices

Write principles carefully

• Keep each principle to one sentence about one idea.
• State the preferred behavior, not just the forbidden one.
• Add principles for helpfulness, not only for safety.
• Test each principle on a handful of prompts and read the revisions.

Start with a small pilot

Run the critique and revision loop on a few hundred prompts. Read the outputs. If revisions are bland or wrongly preachy, change the wording of the principles or the critique prompt before scaling.

Mix data thoughtfully

Blend safety-focused revised answers with ordinary helpful data. Check the ratio against over-refusal tests. A small shift in the ratio can change behavior a lot.

Use human review where stakes are high

Have people review a sample of critiques, revisions and AI labels. Review more heavily in sensitive domains.

Version everything

Keep versioned copies of the constitution, the critique prompts, the labeling prompts and the model checkpoints. When behavior changes, you can trace it.

Test with adversarial prompts

Use red teaming to look for gaps. Add the failures you find to the next round of prompts. Keep a fixed regression set so you can see progress.

Keep run-time defenses

Do not rely on training alone. Add input checks, output checks, logging and rate limits.

Combine with efficient training tools

Parameter-efficient methods make the loop cheaper. Use LoRA adapters with PEFT, and run SFT or preference stages with TRL or Axolotl.

Measure both sides

Common mistakes

• Vague principles. "Be good" gives no guidance. Write concrete rules.
• Too many principles at once. Apply one per round to keep critiques focused.
• Only safety data. The model becomes timid and unhelpful.
• Trusting the critique blindly. Critiques can be wrong or off-topic. Spot-check them.
• Training on drafts instead of revisions. The dataset must hold the final answers.
• No held-out harmless set. Over-refusal goes unnoticed.
• Single-order labeling. Position bias leaks into labels.
• Treating the constitution as complete. New risks appear. Plan to update it.
• Confusing training with enforcement. A trained model can still be tricked. Keep guardrails.
• No audit trail. Without versioned artifacts, you cannot explain why behavior changed.
• Using the policy as its own only judge. Add a second judge or human sampling.

Frequently asked questions

What is Constitutional AI in simple terms?

Constitutional AI is a training method in which a language model learns to follow a written list of principles. The model critiques and rewrites its own answers using those principles, then learns from the improved answers and from AI-made comparisons of answers.

Who wrote the constitution?

The authors of each system do. The original research described a list drawn from sources such as general ethical ideas and published rights texts, but any team can write its own list for its own setting. Some later work has explored gathering public input on the principles.

Is Constitutional AI the same as RLAIF?

Not exactly. RLAIF is the general idea of using AI-generated labels in place of human ones. Constitutional AI is a specific recipe that uses a written list of principles to guide self-revision and AI labeling, so it is one example of RLAIF.

Does it remove the need for human labels?

No. It reduces the number of human labels needed for harmlessness, but people still write the principles, supply or check helpfulness data, and audit results. Human review remains important, especially in sensitive areas.

Can the model really critique itself?

Yes, to a useful degree. Capable models can point out how a draft conflicts with a stated rule and can rewrite it. The quality varies, so critiques should be sampled and checked, and stronger models generally do better.

Does Constitutional AI make a model safe?

It improves behavior on the kinds of cases covered in training, but it does not make a model fully safe. Attackers can still find bypasses, so it should be combined with run-time guardrails, monitoring, access control and regular red teaming.

How is it different from a system prompt with rules?

A system prompt steers behavior at run time and can be overridden or forgotten in long contexts. Constitutional training changes the model weights so the behavior is built in. Many systems use both, since a prompt is easy to change and training is more robust.

Can I use it with small open models?

Yes, in principle. You can run the critique and revision loop with a capable model, then fine-tune a smaller model on the results with tools such as TRL and PEFT. Check that the labeling model is strong enough, since weak critiques give weak data.

How does it relate to tabular data work?

It applies to the language parts of table pipelines, such as column naming, sector labels, imputation policies and anomaly explanations. Written rules can guide and test these behaviors. It does not improve numeric prediction, which depends on models built for tables and on real labels.

How do I know my principles are working?

Test them. Use a set of tricky prompts that target each principle, a set of harmless prompts to check for over-refusal, and human review of a sample. If behavior changes in the intended direction without hurting general ability, the principles are working.

Glossary

• AI feedback: Labels or critiques produced by a model.
• Alignment: Making model behavior match intended goals and values.
• Constitution: A written list of principles guiding model behavior.
• Critique: A short note on how an answer breaks a rule.
• Guardrail: A run-time check on inputs or outputs.
• Harmlessness: The quality of avoiding harmful outputs.
• Helpfulness: The quality of giving useful and correct help.
• Over-refusal: Declining requests that are actually fine.
• Preference model: A model that scores answers by learned preference.
• Principle: One rule inside the constitution.
• Red teaming: Testing a model by trying to make it fail.
• Revision: An improved answer written after a critique.
• RLAIF: Reinforcement learning from AI feedback.
• Self-revision: A model improving its own draft.
• Supervised fine-tuning: Training on example inputs and target outputs.

Summary

Constitutional AI aligns a language model with a written list of principles. In stage one, the model critiques and revises its own drafts against randomly chosen principles, and is fine-tuned on the revisions. In stage two, an AI judge applies the principles to label which of two answers is better, and the model is tuned with a preference model or a direct preference method. The result is a model with more readable, editable behavior rules and fewer human labels, along with better-explained refusals.

Its limits are real: rule quality, judge bias, coverage gaps, over-refusal and the fact that training is not enforcement. Good practice is to write short concrete principles, include helpfulness rules, mix data carefully, audit with humans, red team, version everything and keep run-time guardrails. In table work, written principles help steer semantic labeling, imputation policies, anomaly reports and privacy behavior, while numeric prediction stays with models built for tables. Related pages include RLAIF, RLHF, DPO, TRL, guardrails, and the model pages /model/architecture and /model/benchmarks.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: