3. Reading the results
The report, its verdicts, the chart, and adding repeats to a finished experiment.
When the last search finishes, hillclimb experiment run prints the report. You can print it
again at any time, also while the study is still running:
hillclimb experiment report mine-vs-greedyThe table
One row per setup, for each problem:
| column | what it is |
|---|---|
n | searches finished |
mean, median, spread | the final score across those searches |
wins | repeats in which this setup had the best score |
min→best | minutes until the search found its best candidate |
tokens | tokens the coding agents used |
The score is the one each search would ship: its selected candidate, scored on the holdout split when the problem has one, and on the validation split otherwise. A holdout score is the honest one, because no climber ever saw it.
The verdict
Below the table, every setup is compared with the control, repeat by repeat, and the gap is judged against the noise floor:
mine: +0.012 vs greedy — better beyond noise (0.01); wins 3, loses 0, ties 0wins, loses and ties count the repeats one by one: repeat 1 of mine against repeat 1 of
greedy, and so on. A gap inside the noise floor is reported as within noise, not a result. That is an answer
too: the change made no difference you could measure at this number of repeats. --control
measures against another setup, and --noise-floor tries a different threshold without
running anything again.
--json prints the same summaries as JSON, with a verdict for every comparison (better,
worse, tie, within-noise or unknown), for a script or an agent to read.
The chart
hillclimb chartEach search of the study is a climb on the chart, coloured by its setup, so you can see whether one setup gets to a good score sooner as well as higher.
Adding repeats later
If the result is close, add repeats to the same study instead of starting a new one. The new searches join the old ones in the report and the chart:
hillclimb experiment run mine-vs-greedy --run-id <run-id> --first-repeat 4 --repeats 2That adds repeats 4 and 5 to the run that holds the first three.