Run experiments in CI/CD

Integrate evaluations into your CI/CD pipeline to catch regressions before they reach production. Run evals on every pull request, gate merges on the results, and report scores back to your team automatically.

GitHub Actions

Use the braintrustdata/eval-action to run evaluations on every pull request:

name: Run evaluations

on:
  pull_request:
    branches: [main]

permissions:
  pull-requests: write
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Install dependencies
        run: npm install

      - name: Run evals
        uses: braintrustdata/eval-action@v2
        with:
          api_key: ${{ secrets.BRAINTRUST_API_KEY }}
          runtime: node

Set runtime to node, python, or go to match your project. The action automatically posts a comment with results to the pull request, which requires the pull-requests: write permission shown above.

Other CI systems

For other CI systems, use bt eval directly:

BRAINTRUST_API_KEY=$BRAINTRUST_API_KEY bt eval evals/ --no-input --json

Create an API key under Settings > API keys and set it as BRAINTRUST_API_KEY in your CI environment. Use --no-input to suppress prompts and --json for machine-readable output. Use --first N or --sample N to run a non-final smoke test on pull requests and reserve the full run for merges:

bt eval tests/ --first 20 --no-input --json    # smoke run on PR, non-final
bt eval tests/ --no-input --json               # full run on merge, final

Customize pass/fail with reporters

When you run an experiment, Braintrust logs results to your terminal, and bt eval returns a non-zero exit code if any eval throws an exception. Customize this behavior for CI/CD pipelines to precisely define what constitutes a failure or to report results to different systems. Define custom reporters using Reporter(). A reporter has two functions:

import { Reporter } from "braintrust";

Reporter(
  "My reporter", // Replace with your reporter name
  {
    reportEval(evaluator, result, opts) {
      // Summarizes the results of a single evaluator and returns whatever you
      // want (the full results, a piece of text, or both)
    },

    reportRun(results) {
      // Takes all the results and summarizes them. Return a true or false
      // which tells the process to exit.
      return true;
    },
  },
);

from braintrust import Reporter

def report_eval(evaluator, result, opts):
    # Summarizes the results of a single evaluator and returns whatever you
    # want (the full results, a piece of text, or both)
    pass

def report_run(results):
    # Takes all the results and summarizes them. Return a true or false
    # which tells the process to exit.
    return True

Reporter(
    "My reporter",  # Replace with your reporter name
    report_eval=report_eval,
    report_run=report_run,
)

Any Reporter included among your evaluated files will be automatically picked up by the bt eval CLI command.

If no reporters are defined, the default reporter logs results to the console.
If you define one reporter, it’s used for all Eval blocks.
If you define multiple Reporters, specify the reporter name as an optional third argument to the eval function.

Next steps

Run experiments in code with the SDK
Interpret results from your experiments
Compare experiments to measure improvements
Write scorers to measure quality

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Run experiments in CI/CD

GitHub Actions

Other CI systems

Customize pass/fail with reporters

Next steps

​GitHub Actions

​Other CI systems

​Customize pass/fail with reporters

​Next steps

GitHub Actions

Other CI systems

Customize pass/fail with reporters

Next steps