Experiments

An experiment is an immutable snapshot of an evaluation run: permanently stored, comparable over time, and shareable across your team. Unlike playground runs, which overwrite previous results for fast iteration, experiments preserve exact results so you can measure improvements, catch regressions, and build confidence in your changes.

Create an experiment

Choose the approach that fits your workflow. Each one produces the same kind of experiment, comparable to any other in your project.

Run in code with the SDK. The most flexible path, with full control over your task, data, and scorers, and the one to use when you want evals versioned alongside your application.
Run in the UI, without writing code, from an existing dataset or by promoting a playground run.
Run in CI/CD, to catch regressions automatically on every pull request.

For advanced SDK techniques like trials, hill climbing, attachments, and tracing, see Advanced eval techniques.

Analyze results

Once an experiment finishes, dig into the results:

Interpret results to read the experiment table and drill into individual traces.
Compare experiments to measure improvements and regressions between runs.

Next steps

Run experiments in code with the SDK
Write scorers to measure quality
Build datasets to power your evaluations
Evaluation best practices for reliable, high-signal evals

Iterate in playgrounds

Run experiments in code

⌘I

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Create an experiment

Analyze results

Next steps

​Create an experiment

​Analyze results

​Next steps

Create an experiment

Analyze results

Next steps