Advanced eval techniques

Once you have a basic eval running in code, layer on these techniques to sharpen your scoring signal, evaluate richer inputs, and debug problems. Each is independent. Use the techniques that fit your situation:

Getting noisy or inconsistent scores? Measure variance with trials.
No expected outputs to compare against? Iterate with hill climbing.
Evaluating images, audio, or PDFs? Add attachments.
Eval running slowly or throwing errors? Trace and debug your task.
Iterating locally and don’t want to save runs? Run without uploading results.

These techniques build on the SDK Eval() function. See Run experiments in code to get a basic eval running first.

Measure score variance with trials

Run each input multiple times to measure variance and get more robust scores. Braintrust intelligently aggregates results by bucketing test cases with the same input value:

Eval("My Project", {
  data: myDataset,
  task: myTask,
  scores: [Factuality],
  trialCount: 10, // Run each input 10 times
});

Eval(
    "My Project",
    data=my_dataset,
    task=my_task,
    scores=[Factuality],
    trial_count=10,  # Run each input 10 times
)

package main

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/sdk/trace"

	"github.com/braintrustdata/braintrust-sdk-go"
	"github.com/braintrustdata/braintrust-sdk-go/eval"
)

func main() {
	ctx := context.Background()

	tp := trace.NewTracerProvider()
	defer tp.Shutdown(ctx)
	otel.SetTracerProvider(tp)

	client, err := braintrust.New(tp)
	if err != nil {
		log.Fatal(err)
	}

	evaluator := braintrust.NewEvaluator[string, string](client)

	// Example variables (define your own)
	myDataset := eval.NewDataset([]eval.Case[string, string]{
		{Input: "example input", Expected: "example expected"},
	})
	myTask := eval.T(func(ctx context.Context, input string) (string, error) {
		return "model output", nil
	})
	myScorers := []eval.Scorer[string, string]{
		eval.NewScorer("exact-match", func(ctx context.Context, r eval.TaskResult[string, string]) (eval.Scores, error) {
			score := 0.0
			if r.Output == r.Expected {
				score = 1.0
			}
			return eval.S(score), nil
		}),
	}

	_, err = evaluator.Run(ctx, eval.Opts[string, string]{
		Experiment: "My Project",
		Dataset:    myDataset,
		Task:       myTask,
		Scorers:    myScorers,
		TrialCount: 10, // Run each input 10 times
	})
	if err != nil {
		log.Fatal(err)
	}
}

Braintrust::Eval.run(
  project: "My Project",
  cases: my_dataset,
  task: my_task,
  scorers: my_scorers,
  trial_count: 10  # Run each input 10 times
)

import dev.braintrust.Braintrust;
import dev.braintrust.eval.DatasetCase;
import dev.braintrust.eval.Scorer;
import java.util.function.Function;

class Main {
  public static void main(String... args) {
    var braintrust = Braintrust.get();
    var openTelemetry = braintrust.openTelemetryCreate();

    // Example variables (define your own)
    var myDataset = new DatasetCase[]{
        DatasetCase.of("example input", "example expected")
    };
    Function<String, String> myTask = input -> "model output";
    var myScorers = new Scorer[]{
        Scorer.of("exact_match", (expected, actual) -> expected.equals(actual) ? 1.0 : 0.0)
    };

    var result = braintrust.<String, String>evalBuilder()
        .name("My Project")
        .cases(myDataset)
        .taskFunction(myTask)
        .scorers(myScorers)
        .build()
        .run();

    System.out.println(result.createReportString());
  }
}

using System;
using System.Threading.Tasks;
using Braintrust.Sdk;
using Braintrust.Sdk.Eval;

class Program
{
    static async Task Main(string[] args)
    {
        var braintrust = Braintrust.Sdk.Braintrust.Get();

        // Example variables (define your own)
        var myDataset = new[] {
            new DatasetCase<string, string>("example input", "example expected")
        };
        Func<string, string> myTask = input => "model output";
        var myScorers = new[] {
            new FunctionScorer<string, string>("exact_match", (expected, actual) =>
                actual == expected ? 1.0 : 0.0)
        };

        var eval = await braintrust
            .EvalBuilder<string, string>()
            .Name("My Project")
            .Cases(myDataset)
            .TaskFunction(myTask)
            .Scorers(myScorers)
            .BuildAsync();

        var result = await eval.RunAsync();
        Console.WriteLine(result.CreateReportString());
    }
}

To analyze trial results and compare variance across inputs, see Compare trials.

Override trial count per case

Individual data rows can set their own trial count to override the global default. Use this when a few inputs need extra trials to measure variance, while the rest don’t:

Eval("My Project", {
  data: [
    { input: "easy",   expected: "answer" },                  // uses global trialCount
    { input: "hard",   expected: "answer", trialCount: 5 },   // run 5 times
    { input: "stable", expected: "answer", trialCount: 1 },   // run once
  ],
  task: myTask,
  scores: [Factuality],
  trialCount: 2, // global default
});

from braintrust import Eval, EvalCase

Eval(
    "My Project",
    data=[
        EvalCase(input="easy",   expected="answer"),                  # uses global trial_count
        EvalCase(input="hard",   expected="answer", trial_count=5),   # run 5 times
        EvalCase(input="stable", expected="answer", trial_count=1),   # run once
    ],
    task=my_task,
    scores=[Factuality],
    trial_count=2,  # global default
)

myDataset := eval.NewDataset([]eval.Case[string, string]{
	{Input: "easy", Expected: "answer"},                  // uses global TrialCount
	{Input: "hard", Expected: "answer", TrialCount: 5},   // run 5 times
	{Input: "stable", Expected: "answer", TrialCount: 1}, // run once
})

_, err := evaluator.Run(ctx, eval.Opts[string, string]{
	Experiment: "My Project",
	Dataset:    myDataset,
	Task:       myTask,
	Scorers:    myScorers,
	TrialCount: 2, // global default
})

Per-case values take precedence over the global trialCount / trial_count / TrialCount. If neither is set, the input runs once.

Iterate without expected outputs

Hill climbing lets you improve iteratively without expected outputs by using a previous experiment’s output as the expected for the current run. To enable it, use BaseExperiment() in the data field. Autoevals scorers like Battle and Summary are designed specifically for this workflow.

import { Battle } from "autoevals";
import { Eval, BaseExperiment } from "braintrust";

Eval<string, string, string>(
  "Say Hi Bot", // Replace with your project name
  {
    data: BaseExperiment(),
    task: (input) => {
      return "Hi " + input; // Replace with your task function
    },
    scores: [Battle.partial({ instructions: "Which response said 'Hi'?" })],
  },
);

from autoevals import Battle
from braintrust import BaseExperiment, Eval

Eval(
    "Say Hi Bot",  # Replace with your project name
    data=BaseExperiment(),
    task=lambda input: "Hi " + input,  # Replace with your task function
    scores=[Battle.partial(instructions="Which response said 'Hi'?")],
)

Braintrust automatically picks the best base experiment using git metadata if available or timestamps otherwise, then populates the expected field by merging the expected and output fields from the base experiment. If you set expected through the UI while reviewing results, it will be used as the expected field for the next experiment. To use a specific experiment as the base, pass the name field to BaseExperiment():

import { Battle } from "autoevals";
import { Eval, BaseExperiment } from "braintrust";

Eval<string, string, string>(
  "Say Hi Bot", // Replace with your project name
  {
    data: BaseExperiment({ name: "main-123" }),
    task: (input) => {
      return "Hi " + input; // Replace with your task function
    },
    scores: [Battle.partial({ instructions: "Which response said 'Hi'?" })],
  },
);

from autoevals import Battle
from braintrust import BaseExperiment, Eval

Eval(
    "Say Hi Bot",  # Replace with your project name
    data=BaseExperiment(name="main-123"),
    task=lambda input: "Hi " + input,  # Replace with your task function
    scores=[Battle.partial(instructions="Which response said 'Hi'?")],
)

When hill climbing, use two types of scoring functions:

Non-comparative methods like ClosedQA that judge output quality based purely on input and output without requiring an expected value. Track these across experiments to compare any two experiments, even if they aren’t sequentially related.
Comparative methods like Battle or Summary that accept an expected output but don’t treat it as ground truth. If you score > 50% on a comparative method, you’re doing better than the base on average. Learn more about how Battle and Summary work.

Evaluate images, audio, and PDFs

Braintrust allows you to log binary data like images, audio, and PDFs as attachments. Use attachments in evaluations by initializing an Attachment object in your data:

import { Eval, Attachment } from "braintrust";
import { NumericDiff } from "autoevals";
import path from "path";

function loadPdfs() {
  return ["example.pdf"].map((pdf) => ({
    input: {
      file: new Attachment({
        filename: pdf,
        contentType: "application/pdf",
        data: path.join("files", pdf),
      }),
    },
    // This is a toy example where we check that the file size is what we expect.
    expected: 469513,
  }));
}

async function getFileSize(input: { file: Attachment }) {
  return (await input.file.data()).size;
}

Eval("Project with PDFs", {
  data: loadPdfs,
  task: getFileSize,
  scores: [NumericDiff],
});

import os
from typing import Any, Dict, Iterable

from autoevals import NumericDiff
from braintrust import Attachment, Eval, EvalCase

def load_pdfs() -> Iterable[EvalCase[Dict[str, Any], int]]:
    for filename in ["example.pdf"]:
        yield EvalCase(
            input={
                "file": Attachment(
                    filename=filename,
                    content_type="application/pdf",
                    # The file on your filesystem or the file's bytes.
                    data=os.path.join("files", filename),
                )
            },
            # This is a toy example where we check that the file size is what we expect.
            expected=469513,
        )

def get_file_size(input: Dict[str, Any]) -> int:
    return len(input["file"].data)

# Our evaluation uses a NumericDiff scorer to check the file size.
Eval(
    "Project with PDFs",
    data=load_pdfs(),
    task=get_file_size,
    scores=[NumericDiff],
)

You can also store attachments in a dataset for reuse across multiple experiments. After creating the dataset, reference it by name in an eval. The attachment data is automatically downloaded from Braintrust when accessed:

import { NumericDiff } from "autoevals";
import { initDataset, Eval, ReadonlyAttachment } from "braintrust";

async function getFileSize(input: {
  file: ReadonlyAttachment;
}): Promise<number> {
  return (await input.file.data()).size;
}

Eval("Project with PDFs", {
  data: initDataset({
    project: "Project with PDFs",
    dataset: "My PDF Dataset",
  }),
  task: getFileSize,
  scores: [NumericDiff],
});

from autoevals import NumericDiff
from braintrust import Eval, init_dataset

def get_file_size(input: Dict[str, Any]) -> int:
    """Download the attachment and get its length."""
    return len(input["file"].data)

Eval(
    "Project with PDFs",
    data=init_dataset("Project with PDFs", "My PDF Dataset"),
    task=get_file_size,
    scores=[NumericDiff],
)

To forward an attachment to an external service like OpenAI, obtain a signed URL instead of downloading the data directly:

import { initDataset, wrapOpenAI, ReadonlyAttachment } from "braintrust";
import { OpenAI } from "openai";

const client = wrapOpenAI(
  new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  }),
);

async function main() {
  const dataset = initDataset({
    project: "Project with images",
    dataset: "My Image Dataset",
  });
  for await (const row of dataset) {
    const attachment: ReadonlyAttachment = row.input.file;
    const attachmentUrl = (await attachment.metadata()).downloadUrl;
    const response = await client.chat.completions.create({
      model: "gpt-5-mini",
      messages: [
        {
          role: "system",
          content: "You are a helpful assistant",
        },
        {
          role: "user",
          content: [
            { type: "text", text: "Please summarize the attached image" },
            { type: "image_url", image_url: { url: attachmentUrl } },
          ],
        },
      ],
    });
    const summary = response.choices[0].message.content || "Unknown";
    console.log(
      `Summary for file ${attachment.reference.filename}: ${summary}`,
    );
  }
}

main();

from braintrust import init_dataset, wrap_openai
from openai import OpenAI

openai = wrap_openai(OpenAI(api_key=os.environ["OPENAI_API_KEY"]))

def main():
    dataset = init_dataset("Project with images", "My Image Dataset")
    for row in dataset:
        attachment = row["input"]["file"]
        attachment_url = attachment.metadata()["downloadUrl"]
        response = openai.chat.completions.create(
            model="gpt-5-mini",
            messages=[
                {"role": "system", "content": "You are a helpful assistant"},
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Please summarize the attached image"},
                        {"type": "image_url", "image_url": {"url": attachment_url}},
                    ],
                },
            ],
        )
        summary = response.choices[0].message.content or "Unknown"
        print(f"Summary for file {attachment.reference['filename']}: {summary}")

main()

Trace and debug your eval tasks

Add detailed tracing to your evaluation task functions to measure performance and debug issues. Each span in the trace represents an operation like an LLM call, database lookup, or API request.

Use wrapOpenAI/wrap_openai to automatically trace OpenAI API calls. See Trace LLM calls for details.

Each call to experiment.log() creates its own trace. Do not mix experiment.log() with tracing functions like traced(). Doing so creates incorrectly parented traces.

Wrap task code with traced() to log incrementally to spans. This example progressively logs input, output, and metrics:

import { Eval, traced } from "braintrust";

async function callModel(input: string) {
  return traced(
    async (span) => {
      const messages = { messages: [{ role: "system", text: input }] };
      span.log({ input: messages });

      // Replace this with a model call
      const result = {
        content: "China",
        latency: 1,
        prompt_tokens: 10,
        completion_tokens: 2,
      };

      span.log({
        output: result.content,
        metrics: {
          latency: result.latency,
          prompt_tokens: result.prompt_tokens,
          completion_tokens: result.completion_tokens,
        },
      });
      return result.content;
    },
    {
      name: "My AI model",
    },
  );
}

const exactMatch = (args: {
  input: string;
  output: string;
  expected?: string;
}) => {
  return {
    name: "Exact match",
    score: args.output === args.expected ? 1 : 0,
  };
};

Eval("My Evaluation", {
  data: () => [
    { input: "Which country has the highest population?", expected: "China" },
  ],
  task: async (input, { span }) => {
    return await callModel(input);
  },
  scores: [exactMatch],
});

from braintrust import Eval, current_span, traced

@traced
async def call_model(input):
    messages = dict(
        messages=[
            dict(role="system", text=input),
        ]
    )
    current_span().log(input=messages)

    # Replace this with a model call
    result = {
        "content": "China",
        "latency": 1,
        "prompt_tokens": 10,
        "completion_tokens": 2,
    }
    current_span().log(
        output=result["content"],
        metrics=dict(
            latency=result["latency"],
            prompt_tokens=result["prompt_tokens"],
            completion_tokens=result["completion_tokens"],
        ),
    )
    return result["content"]

async def run_input(input):
    return await call_model(input)

def exact_match(input, expected, output):
    return 1 if output == expected else 0

Eval(
    "My Evaluation",
    data=[dict(input="Which country has the highest population?", expected="China")],
    task=run_input,
    scores=[exact_match],
)

This creates a span tree you can visualize in the UI by clicking on each test case in the experiment.

Troubleshooting

Evaluations running slowly with maxConcurrency

If your evaluations are slower than expected when using maxConcurrency, you may be on an older SDK version that flushes logs after every single task completion. Upgrade to TypeScript SDK v3.3.0+ for up to an 8x performance improvement. The SDK now uses byte-based backpressure for better flushing performance.You can tune the flush threshold with the BRAINTRUST_FLUSH_BACKPRESSURE_BYTES environment variable. See Tune performance for all available configuration options.

Task function throws an exception during eval (C# SDK v0.2.2+)

When the task function throws, the C# eval framework catches the exception, records it on the task span and root span (with ActivityStatusCode.Error), and calls ScoreForTaskException on every scorer instead of Score. The eval continues — no cases are skipped.By default, ScoreForTaskException returns a single score of 0.0. Override it on your IScorer to return a custom fallback score, return an empty list to omit scoring for that case, or re-throw to abort the eval.

#skip-compile

using Braintrust.Sdk.Eval;

sealed class MyScorer : IScorer<string, string>
{
    public string Name => "my_scorer";

    public Task<IReadOnlyList<Score>> Score(TaskResult<string, string> taskResult)
    {
        var matches = taskResult.Result == taskResult.DatasetCase.Expected;
        return Task.FromResult<IReadOnlyList<Score>>([new Score(Name, matches ? 1.0 : 0.0)]);
    }

    // Called instead of Score() when the task function threw.
    // Return [] to skip recording a score; throw to abort the eval.
    public Task<IReadOnlyList<Score>> ScoreForTaskException(
        Exception taskException,
        DatasetCase<string, string> datasetCase)
    {
        // Distinguish between expected and unexpected failures
        if (taskException is TimeoutException)
            return Task.FromResult<IReadOnlyList<Score>>([new Score(Name, 0.0)]);

        return Task.FromResult<IReadOnlyList<Score>>([]); // skip scoring
    }
}

The task span and root eval span both receive an OTel exception event with exception.type, exception.message, and exception.stacktrace attributes, visible in any OTel-compatible backend connected to Braintrust.

Scorer throws an exception during eval (C# SDK v0.2.2+)

When a scorer’s Score method throws, the exception is recorded on that scorer’s span (with ActivityStatusCode.Error and an OTel exception event) and ScoreForScorerException is called as a fallback. Other scorers continue running unaffected.By default, ScoreForScorerException returns a single score of 0.0. Override it to return a custom fallback, return an empty list to omit the score, or re-throw to abort the eval.

#skip-compile

using Braintrust.Sdk.Eval;

sealed class MyScorer : IScorer<string, string>
{
    public string Name => "my_scorer";

    public Task<IReadOnlyList<Score>> Score(TaskResult<string, string> taskResult)
    {
        // ... scoring logic that might throw
        throw new InvalidOperationException("unexpected output format");
    }

    // Called when Score() throws. Other scorers are not affected.
    // Return [] to skip recording a score; throw to abort the eval.
    public Task<IReadOnlyList<Score>> ScoreForScorerException(
        Exception scorerException,
        TaskResult<string, string> taskResult)
    {
        return Task.FromResult<IReadOnlyList<Score>>([new Score(Name, 0.0)]);
    }
}

Score spans are named score:<scorer_name> (e.g. score:my_scorer), making individual scorer traces distinguishable in Braintrust and any connected OTel backend.

Run without uploading results

Sometimes you want to run your evaluation locally without creating an experiment in Braintrust — while iterating on a new scorer, wiring up a new eval pipeline, or running in an environment without a Braintrust API key. Your tasks and scorers still run and print a summary to your terminal; results just aren’t uploaded.

TypeScript
Python

Via the CLI:

bt eval --no-send-logs my_eval.eval.ts

Or in code:

Eval("My Project", {
  data: ...,
  task: ...,
  scores: [...],
  noSendLogs: true,
});

Via the CLI:

bt eval --no-send-logs my_eval.py

Or in code:

Eval(
    "My Project",
    data=...,
    task=...,
    scores=[...],
    no_send_logs=True,
)

Next steps

Interpret results from your experiments
Compare experiments to measure improvements
Test complex agents to connect custom code to the playground
Write scorers to measure quality
Evaluation best practices for reliable, high-signal evals

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Advanced eval techniques

Measure score variance with trials

Override trial count per case

Iterate without expected outputs

Evaluate images, audio, and PDFs

Trace and debug your eval tasks

Troubleshooting

Run without uploading results

Next steps

​Measure score variance with trials

​Override trial count per case

​Iterate without expected outputs

​Evaluate images, audio, and PDFs

​Trace and debug your eval tasks

​Troubleshooting

​Run without uploading results

​Next steps

Measure score variance with trials

Override trial count per case

Iterate without expected outputs

Evaluate images, audio, and PDFs

Trace and debug your eval tasks

Troubleshooting

Run without uploading results

Next steps