# Regression boards

A regression board is one permanent suite of probes that checks every known requirement and past bug after each change. Learn how to build one that stays honest.

## Overview

A regression board is one permanent, growing suite of probes that checks every requirement and every bug ever fixed in a system. Each line on the board sends a real request, compares the response with an expected result and prints pass or fail. After each phase of changes, the whole board runs, not just the lines for the latest work. A fix counts as done only when it has its own line on the board, so it can never quietly break again.

"Regression board" is a descriptive name rather than a formal standard. It applies the long-established practice of regression testing to the way modern AI-backed systems are actually checked: by external audits, specifications and incident reports, measured end to end through public interfaces. The board turns each of those findings into a permanent, objective check.

Think of a pilot's pre-flight checklist. It does not change with the pilot's mood. It covers everything that has ever mattered, and every item is ticked before take-off. When an incident reveals a new risk, a new item is added for every future flight. A regression board is that checklist for software and AI systems.

## What it is

A regression board is defined by a few properties more than by any tool.

| Property | What it means | Why it matters |
|---|---|---|
| One board per system | A single script or suite, not scattered checks | Nothing is forgotten in an old folder |
| One line per requirement or fixed defect | Each item has its own probe with an id | Closed items stay closed |
| Expected result on every line | Status code, fields, values or a score threshold | Pass or fail is objective |
| Real interfaces | Probes call the system the way users and auditors do | Tests what is shipped, not internals |
| Fixtures from the real schema | Test data generated from current definitions | No false failures from stale hard-coded columns |
| Fresh subjects | Each probe uses clean test objects | Probes do not interfere with each other |
| Rate-aware execution | Waits and retries on throttling, pauses between probes | Avoids fake failures from rate limits |
| Pass and fail counter | A summary at the end of every run | Progress is visible at a glance |
| Full run at phase end | The whole board runs after each batch of changes | Catches new fixes that break old ones |

## Where it came from

Regression testing, re-running earlier tests after a change to check that nothing that used to work has broken, is one of the oldest practices in software engineering. Its importance grew with automated test suites and continuous integration, which became widespread with Extreme Programming in the late 1990s and later with hosted build services. A common rule from that tradition is that every bug fix comes with a test that would have caught it.

AI systems and API platforms add new pressures. Behaviour depends on models, prompts, data and configuration as well as code. Requirements often arrive as specification documents, external audit reports and incident lists rather than as unit-test tickets. Many teams found that fixes passed their own internal tests but failed when an auditor re-tested with a different request, or that a later fix silently re-opened an earlier finding.

The regression board pattern answers this by organising checks around requirements and findings, measuring each one the way its source describes, and treating a line on the board as the definition of "closed". It complements [evals](/tools/evals), which measure the quality of AI outputs on datasets, by covering contracts, error handling, security behaviour and every past defect end to end.

## How it works

### 1. Take a full inventory

When a specification, audit report or incident list arrives, list every item, including sub-requirements and notes, before starting work. Each item gets an id that the board will use.

### 2. Write the probe the way the source describes

Each probe reproduces the exact request the requirement or finding describes: the same endpoint, method, headers, body shape and sequence of calls. If a finding says "a single update followed by a read returns the old value", the probe does exactly that, not a convenient variant.

### 3. Generate fixtures from the schema

Test data is created from the system's current schema or contract at run time, rather than from hard-coded ids and column names that drift; see [schema drift detection](/tools/schema-drift-detection). Each probe works on fresh test objects so earlier probes cannot affect it.

### 4. Check every surface

A defect class usually appears in several places: a list response, a single-item response, an embedded copy inside a report, a stored value in a database or file name, a log line and a replay path. The board has a line for each surface, not just the one first noticed.

### 5. Check the reverse direction

After a fix that removes or hides something, add a probe that confirms it is still present where it is required. Over-correction is a regression too.

### 6. Run politely and report clearly

The runner respects rate limits, waits and retries on throttling responses, and pauses between probes. Each line prints the observed result, the expected result and pass or fail. The run ends with a counter.

A minimal probe in a shell runner might look like this:

```bash
probe() {  # id, expected_status, curl arguments...
  local id=$1 want=$2; shift 2
  for attempt in 1 2 3; do
    got=$(curl -s -o /dev/null -w '%{http_code}' "$@")
    [ "$got" != 429 ] && break
    sleep $((attempt * 5))
  done
  if [ "$got" = "$want" ]; then pass=$((pass+1)); echo "PASS $id $got"
  else fail=$((fail+1)); echo "FAIL $id got $got want $want"; fi
  sleep 1
}
```

### 7. Run the whole board at the end of each phase

Changes are made in batches. At the end of each batch the entire board runs, because new patches can break old lines. Only a full green run closes the phase.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Regression | Something that used to work and no longer does |
| Probe | One automated check with a request and an expected result |
| Line | A probe's place on the board, tied to a requirement id |
| Fixture | Test data and objects a probe needs |
| Schema-derived fixture | Test data generated from the current schema |
| Surface | One place where a behaviour is visible, such as a list or a log |
| Over-correction | A fix that removes behaviour where it was needed |
| Rate guard | Logic that waits and retries on throttling |
| Full run | Executing every line on the board |
| Release gate | A rule that blocks release unless the board passes |
| Flaky probe | A probe that passes or fails without a real change |

## A worked example

This is an illustrative scenario. An API platform receives an external audit with 34 main findings and 86 sub-requirements.

1. **Inventory.** The team lists all 86 sub-requirements with ids, including notes and the auditor's route-by-route walkthrough.
2. **Probes.** Each sub-requirement becomes a line, written with the exact request sequence the auditor described. Several findings about naming become five lines each: list, detail, report body, stored file name and log.
3. **First run.** Before any fix, the board shows 31 pass and 55 fail. This confirms the probes reproduce the findings.
4. **Batch one.** Fixes for error formats and pagination are made. The full board runs: 58 pass, 28 fail. One earlier pass, an error-envelope check, now fails because the pagination change returns a bare message on invalid cursors.
5. **Repair.** The error envelope is fixed in the pagination path too, and a new line is added for that surface.
6. **Rate limits.** An early run showed 20 failures that were all throttling responses. The runner now waits and retries, and those fake failures disappear.
7. **Close.** After three batches the full board shows 92 of 92 lines passing, including the added surface lines. The board becomes the starting point for the next audit.

## Strengths and limitations

**Strengths:**

- Prevents fixed defects from quietly returning.
- Makes "done" objective and verifiable by anyone.
- Creates shared truth between engineers, auditors and product owners.
- Turns every incident and finding into permanent protection.
- Catches cross-effects between fixes through full runs.

**Limitations:**

- Boards grow and need maintenance as the system changes.
- Poorly designed probes cause false failures and erode trust.
- End-to-end probes are slower than unit tests.
- A board checks known requirements, not unknown ones.
- Probes against shared environments need care to avoid side effects.

## How it compares with alternatives

| Approach | Coverage | Gap |
|---|---|---|
| Ad hoc manual checks | Whatever someone remembers | Inconsistent and unrepeatable |
| Unit tests | Code-level behaviour | Miss integration and contract issues |
| [Evals](/tools/evals) | Quality of AI outputs on datasets | Not focused on contracts and past defects |
| [Data contract validation](/tools/data-contract-validation) | Shape and rules of data | Does not test behaviour end to end |
| Regression board | Every known requirement and fixed defect, end to end | Needs upkeep and careful probe design |

## Practical guidance and best practices

1. Take a full inventory of every item in a report or specification before starting work.
2. Measure each item with the request and data the source describes, not your own convenient fixture.
3. Check every surface a fix touches: list, detail, embedded copies, stored values, logs and replay paths.
4. After a removal-type fix, add a probe that confirms the behaviour still exists where it is needed.
5. Do not close an item with an argument; meet the full request or record exactly what remains.
6. Add the line to the board before calling an item closed.
7. Generate fixtures from the schema and use fresh test subjects.
8. Build in rate guards and pauses so throttling does not produce fake failures.
9. Run the entire board at the end of every phase, and use it as a release gate, for example in [CI/CD](/tools/gitlab-ci-cd).

## Common mistakes

- Testing a fix with a fixture you designed, which passes while the reported case still fails.
- Fixing one surface of a defect and missing the stored value or the log.
- Running only the new lines and missing a fix that broke an old one.
- Failing to keep [traces](/tools/tracing) or logs of failed probes, so nobody can see why a line went red.
- Hard-coding ids and columns that go stale and cause false failures.
- Ignoring throttling, so a fast run reports dozens of fake failures.
- Calling an item closed without adding its line.

## Frequently asked questions

### What is a regression board?

It is a single permanent suite of probes that covers every requirement and every fixed defect of a system, each with an expected result. The whole board runs after each phase of changes, and an item counts as closed only when its line passes.

### How is it different from a normal test suite?

It is organised around requirements, audit findings and incidents rather than code modules. It uses real end-to-end requests, measures each item the way its source describes and makes a board line the definition of closure.

### Why run the whole board instead of only the new lines?

Because new fixes often break older ones in ways nobody predicts. Only a full run reveals these cross-effects before release.

### How do regression boards relate to evals?

Evals measure the quality of AI outputs, such as accuracy or faithfulness, across datasets, often with graders such as [LLM-as-judge](/tools/llm-as-judge). Regression boards check contracts, behaviours and past defects end to end. Many teams use both, and failing eval cases can also become board lines.

### How do you avoid false failures?

Generate fixtures from the current schema, use fresh test objects for each probe, respect rate limits with waits and retries, and keep probes independent of each other. Investigate any flaky probe at once.

### Does the board ever shrink?

Lines are removed only when a requirement is deliberately retired, and that removal should itself be reviewed. Otherwise the board grows with every finding.

## Glossary

- Regression: a return of a previously fixed problem.
- Probe: one automated check on the board.
- Expected result: the outcome that counts as a pass.
- Fixture: data prepared for a probe.
- Surface: a place where behaviour can be observed.
- Rate guard: waiting and retrying on throttling.
- Full run: running every line.
- Release gate: a pass requirement before release.
- Inventory: the complete list of items from a source document.

## Summary

A regression board keeps a system honest. Every requirement, audit finding and fixed defect becomes a permanent probe with an expected result, measured the way its source describes, across every surface where it appears. The board uses schema-derived fixtures and fresh subjects, respects rate limits, prints a clear pass and fail counter, and runs in full at the end of every phase. It builds on the long tradition of regression testing and complements evals, turning each fix into a lasting guarantee.