Overview

A search tells you how well one climber did on one problem, once. An experiment tells you whether one setup is better than another. It runs each setup several times on the same problems, under the same budget, and compares them against how much a score moves by chance alone.

You need one whenever the question is "does this help?": your own climber against greedy, one model against another, memory on or off, a new prompt against the old one.

What an experiment is

You write the comparison down once, in a small YAML file under experiments/. It names:

• the problems to compare on,
• the experiments: each one a setup with a name, written as settings that differ from the folder's defaults. The first one is the control the others are measured against,
• how many repeats each setup gets, and the budget of every search,
• the noise floor: the smallest difference that counts as real.

hillclimb runs every setup on every problem as many times as you asked, each as an ordinary search, and tags each search with its setup and repeat. The report then puts the setups side by side. A whole spec run like this is called a study.

What an experiment is not

• Not a single search with a better result. One search per setup is an anecdote: a coding agent writes different code every time, so two runs of the same setup can land far apart. Repeats are what turn a difference into a result.
• Not a race in one run. Starting two climbers side by side with --climber A --climber B is a quick look and is covered in 1. A quick comparison, but it has one repeat and no noise floor.
• Not tuning a solution. Trying parameter values on a candidate is the climber's tuner, inside one search. An experiment compares whole setups, across searches.
• Not free. Every search in it is a real search, with real coding agent calls. The number of repeats is your cost dial.