# NeMo Guardrails

NeMo Guardrails is NVIDIA's open-source toolkit for adding programmable input, output, dialogue and retrieval rails to LLM applications using the Colang language.

## Overview

NeMo Guardrails is an open-source toolkit from NVIDIA for adding programmable guardrails, called rails, to applications built on large language models (LLMs). A rail is a rule that controls some part of the conversation: what inputs are accepted, which topics the assistant will discuss, how it responds to certain requests, what retrieved text it may use, which tools it may call and what output it may return.

The toolkit sits between your application and the model. Instead of calling the model directly, your code calls the guardrails layer, which applies the configured rails, calls the model when needed and returns the final answer. Rails are written in a mix of YAML configuration, Python actions and Colang, a small language designed for describing dialogue flows.

An everyday analogy: a receptionist with a written procedures manual. Some visitors are turned away at the door, some requests follow a fixed script ("please show your ID first"), some questions are passed to an expert, and anything said on the way out is checked against the rules. The model is the expert; NeMo Guardrails is the receptionist and the manual.

## What it is

The toolkit supports five types of rails.

| Rail type | Applied to | Can do |
|---|---|---|
| Input rails | The user's message | Reject it, or alter it, for example by masking sensitive data |
| Dialogue rails | The flow of the conversation | Choose a predefined response, call an action, or let the model generate the next step |
| Retrieval rails | Chunks retrieved in a RAG application | Reject or alter chunks before they reach the prompt |
| Execution rails | Inputs and outputs of custom actions (tools) | Check what tools receive and return |
| Output rails | The model's response | Reject or alter it before it reaches the user |

Around these sit the other building blocks:

| Part | Role |
|---|---|
| `config.yml` | Models, active rails and settings |
| Colang files (`.co`) | Dialogue flows, canonical user and bot messages |
| `actions.py` | Custom Python actions called from flows |
| Guardrails library | Built-in rails: self-check moderation, fact-checking, hallucination detection, jailbreak and injection detection, NVIDIA safety models and third-party integrations |
| `LLMRails` | The Python object that runs a configuration |
| CLI and server | `nemoguardrails chat`, `server`, `evaluate` and an actions server |

The project is released under the Apache 2.0 licence and supports Python 3.10 to 3.13. A configuration with no rails simply forwards requests to the model.

## Where it came from

NVIDIA released NeMo Guardrails as open source in April 2023, when the first version appeared on PyPI. In October 2023 the team published the paper "NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails" by Traian Rebedea and colleagues, which describes the design and an evaluation.

Since then the project has grown steadily:

- Colang 2.0 was introduced as a more expressive successor to the original language. Both versions are supported, and Colang 1.0 remains the default.
- The built-in library expanded with self-check rails, jailbreak heuristics, fact-checking, sensitive data detection and integrations with external moderation services.
- NVIDIA added its own safety models, such as content safety and topic safety classifiers, as rails.
- A server mode exposes an OpenAI-style chat completions endpoint, and there is an optional LangChain integration.
- The repository now lives under the NVIDIA-NeMo organisation; version 0.24.1 was released in September 2026.

## How it works

### 1. Write a configuration

A configuration folder holds `config.yml`, Colang files and optional Python. The YAML names the main model and lists the active input and output rails, for example a jailbreak check and sensitive data masking on input, and fact-checking on output.

### 2. Define dialogue flows in Colang

Colang describes canonical forms of messages and the flows that connect them. A canonical form is a short label for an intent, such as "user express greeting", with example phrases.

```colang
define user ask about politics
  "Who should I vote for?"
  "What do you think of the government?"

define bot refuse politics
  "I can only help with questions about your account."

define flow politics
  user ask about politics
  bot refuse politics
```

### 3. Run a request

Your code loads the configuration into `LLMRails` and calls `generate` with chat messages, in the same format as common chat APIs. For each turn the runtime typically:

1. Runs input rails. If one rejects the message, processing stops with a safe reply.
2. Maps the user message to a canonical form. It compares the message with example phrases using embeddings and, if needed, asks the model.
3. Finds a matching flow. If a flow defines the next step, the runtime follows it, calling actions or using a predefined bot message.
4. If no flow matches, asks the model to generate the next step or the response, guided by similar examples.
5. Runs retrieval and execution rails on any retrieved chunks or tool calls along the way.
6. Runs output rails on the final message, which may block or edit it.

### 4. Use built-in rails

Many common protections need only a line of YAML: self-check prompts that ask the model whether an input or output breaks a policy, heuristics for jailbreak attempts, fact-checking against retrieved evidence, PII masking through Presidio, and calls to safety classifiers.

### 5. Deploy

You can embed the Python API in your application, or run `nemoguardrails server` to expose configurations over HTTP. A separate actions server can run custom actions in their own process.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Rail | A programmable rule controlling one part of the interaction |
| Colang | NVIDIA's language for defining dialogue flows and rails |
| Canonical form | A short intent label for a user or bot message |
| Flow | A sequence of canonical messages and actions |
| Action | A Python function called from a flow |
| Input rail | Check or transformation on the user's message |
| Output rail | Check or transformation on the model's reply |
| Dialogue rail | Rule that steers the conversation path |
| Retrieval rail | Check on retrieved chunks in RAG |
| Execution rail | Check on tool inputs and outputs |
| Self-check | Asking the model to judge whether a message breaks a policy |
| `LLMRails` | Runtime object that applies a configuration |

## A worked example

This is an illustrative scenario.

A bank builds an assistant that answers questions about card fees and branch hours. It must not give investment advice, must not discuss other customers and must hand off disputes to staff.

1. In `config.yml`, they enable a jailbreak check and sensitive data masking on input, and a self-check output rail with a short policy.
2. In Colang, they define canonical forms for "ask for investment advice", "ask about another customer" and "open a dispute", each with eight to ten example phrases.
3. Flows map the first two to fixed refusals, and the dispute form to an action that creates a ticket and returns a reference number.
4. A retrieval rail drops any retrieved chunk tagged as internal-only before it reaches the prompt.
5. They test with 300 prompts: 200 normal, 60 off-topic and 40 jailbreak attempts. Normal questions get answers in 196 cases; off-topic requests are refused in 57; jailbreak attempts are stopped in 35.
6. They review the misses, add example phrases and a stricter output policy, and re-run. Response time rises by around 300 ms per turn because of the extra model calls, which they reduce by moving some checks to a small classifier.

## Strengths and limitations

**Strengths:**

- Covers the whole interaction: input, dialogue, retrieval, tools and output.
- Dialogue rails can enforce fixed procedures such as authentication or hand-offs.
- Large built-in library of rails and integrations.
- Works with many model providers, as a library or a server.
- Apache 2.0 licence, backed by NVIDIA.

**Limitations:**

- Colang is a new language to learn, and two versions coexist.
- Self-check and generation steps add model calls, latency and cost.
- Intent matching can misclassify unusual phrasing.
- Rails reduce risk but cannot guarantee safety against determined attackers.
- Complex flows can become hard to test and maintain.

## How it compares with alternatives

| Tool | Approach | Strength | Consider it when |
|---|---|---|---|
| NeMo Guardrails | Programmable rails and Colang flows | Conversation control and topic limits | You need to steer whole dialogues |
| [Guardrails AI](/tools/guardrails-ai) | Validators on inputs and outputs | Output validation and structured data | You need field-level checks with fix or re-ask |
| [Presidio](/tools/presidio) | PII detection and anonymisation | Privacy controls | Personal data is the main concern |
| [Prompt injection defence](/tools/prompt-injection-defense) | Layered design patterns | Architecture-level protection | You handle untrusted content and tools |
| [Promptfoo](/tools/promptfoo) | Testing and red-teaming | Finds weaknesses before launch | You want to test rails, not run them |

## Practical guidance and best practices

1. Start with a small set of input and output rails before writing complex flows.
2. Write clear, specific policies for self-check rails.
3. Give each canonical form several varied example phrases, including informal ones.
4. Use dialogue rails for fixed procedures; let the model handle open questions.
5. Measure latency per rail and replace slow model-based checks with classifiers where possible.
6. Build a test set of normal, off-topic and adversarial prompts and run it on every change.
7. Use retrieval rails when retrieved documents may contain sensitive or untrusted text.
8. Log rail decisions so you can review false refusals and misses.

## Common mistakes

- Turning on many self-check rails and being surprised by latency and cost.
- Writing canonical forms with only one or two examples.
- Relying on rails alone, without access control on tools and data.
- Mixing Colang 1.0 and 2.0 syntax in one configuration.
- Never measuring how often legitimate questions are refused.

## Frequently asked questions

### What is Colang?

Colang is a modelling language created for NeMo Guardrails to describe dialogue flows. It has a Python-like syntax for defining canonical user and bot messages and the flows between them. Versions 1.0 and 2.0 are supported, with 1.0 as the default.

### Does NeMo Guardrails only work with NVIDIA models?

No. It supports many model providers, including hosted APIs and open models. NVIDIA's own safety models are optional rails.

### How do input and output rails differ from dialogue rails?

Input and output rails check or change individual messages. Dialogue rails control the path of the conversation, deciding whether to use a fixed reply, call an action or let the model generate.

### Can it protect against jailbreaks?

It includes jailbreak and injection detection rails that reduce risk. No rail stops every attack, so combine them with testing, tool permissions and monitoring.

### Can I run it as a service?

Yes. `nemoguardrails server` loads configurations from a folder and exposes a chat completions endpoint over HTTP.

## Glossary

- LLM: a large language model.
- RAG: retrieval-augmented generation, answering with retrieved documents.
- Jailbreak: an input crafted to make a model ignore its rules.
- Intent: what a user is trying to do with a message.
- Moderation: checking content against a policy.
- Hallucination: confident output that is not supported by facts.
- Actions server: a separate process that runs custom actions.

## Summary

NeMo Guardrails is NVIDIA's open-source toolkit for programmable rails around LLM applications. It applies input, dialogue, retrieval, execution and output rails, configured with YAML, Python and the Colang flow language, and includes a broad library of built-in checks. It is strongest when you need to keep an assistant on topic and follow fixed procedures. Start small, measure latency and false refusals, test with adversarial prompts, and use it as one layer among several defences. See [guardrails](/tools/guardrails) and [red-teaming](/tools/red-teaming) for the wider practice.