2. Writing an experiment

An experiment is a YAML file in your hillclimb folder's experiments/ directory. This one asks whether a selector policy of your own beats greedy's on circle-packing:

# experiments/mine-vs-greedy.yaml
problems: [circle-packing]
repeats: 3
budget: 30m
noise_floor: 0.01
experiments:
  greedy:                                 # the control
    climber: greedy
  mine:                                   # one module swapped
    climber: greedy
    climber.selector_policy: climbers/best_so_far.py

hillclimb experiment run mine-vs-greedy --dry-run   # list the searches, start nothing
hillclimb experiment run mine-vs-greedy

That is two setups on one problem, three times each: six searches of 30 minutes.

The setups

Each entry under experiments: is one setup, written as the settings that differ from your folder's defaults. The keys are the same dotted settings --set takes, so anything in runs/config.yaml can be the thing you compare:

experiments:
  greedy:     {climber: greedy}
  openevolve: {climber: openevolve}
  no-memory:  {climber: greedy, learning.enabled: false}
  opus:       {climber: greedy, model: opus}

climber: picks a whole climber. climber.<module> changes one module of it and keeps the rest, which is how you compare your own selector policy, operator or tuner against the built-in one. The first setup is the control: every other setup is measured against it.

Repeats and budget

repeats is how many searches each setup gets on each problem, and budget is the clock of each search. Repeats are what make the result trustworthy, and they are also what it costs: the study runs setups × problems × repeats real searches. Three is a reasonable start. Both can be changed on the command line without editing the file (--repeats 5, --budget 10m).

The noise floor

Two searches of the same setup never score exactly the same. The noise floor is the smallest difference between two setups that you are willing to call real. Measure it on your problem first:

hillclimb verify circle-packing --repeat 5

The spread it prints is how much the score moves on its own. Put a value at least that large in noise_floor; any gap smaller than it is reported as noise.

One after another, or at once

By default the searches run one after another, repeat by repeat, with the setups taking turns inside each repeat. Use that whenever the setups share state, such as the folder's memory, so that every setup sees it at the same point.

When the setups share nothing, for example two climbers with memory off, they can run at the same time:

schedule: parallel
max_concurrent: 4        # searches alive at once

Always set max_concurrent. Without it every search starts at once, and the ones beyond the machine's coding agent slots spend their budget waiting for a slot.

The same starting point

Searches start from the problem's baseline. To start every search from the same solution of your own instead, name it once:

seed_from: seeds/start.py

Every search in the study then starts from that file, so the setups are compared from identical code. The dry run prints the file it resolved and its checksum.