1. Problem structure
A problem is a folder, and a problem is its verifier.
A problem is a folder, and a problem is its verifier. Users define new problems without changing Python code:
problems/my-problem/
├── problem.yaml
├── description.md
├── verifier.sh # the contract: exit 0 = valid, write the score
└── data/ # optional runtime inputsA problem scores its solutions in one of two shapes:
- A verifier,
verifier.sh, that runs the solution and then its scorer. hillclimb runs onlyverifier.sh; averify.pyis just the scorer it calls. It is the simplest to read, and 2. A simple verifier is a whole problem in this shape. - Two steps,
score:inproblem.yaml(andrun:to call a function the solution defines). hillclimb runs the solution and the scorer in a sandbox each, so the scorer can keep data the solution never sees, and no bash is needed, on Windows too.hillclimb problem newand the bundledknapsackproblem are in this shape; see a scorer of its own.
Minimum problem.yaml:
metric: my-score
higher_is_better: true # true or false, unquotedEverything else is optional:
problem_id: my-problem # default: the folder's name
description: description.md # the default
verifier: verifier.sh # the default
score: score.py # instead of a verifier: the scorer of a two-step problem
run: run.py # with score: the problem's own runner for the solution
time_limit_s: 60 # with score: a solution that runs longer is a failed attempt
private: [hidden] # with score: files only the scorer may read
holdout: true # engine also runs `verifier.sh --holdout`
holdout_inputs: [holdout] # with score: inputs only the holdout split reads
contract: contract.md # what solution.py must be/do (prompt section)
baseline: 0.5 # scored at t=0 as the floor candidate; numeric values also become chart lines
# or a file: `baseline: baseline.py`, scored at t=0 as c000
baseline_files: # or files the floor is scored on as they are
submission.csv: sample_submission.csv
chart_baselines: # optional named horizontal lines in `hillclimb chart`
previous best: 0.73
output_artifacts: [submission.csv] # files of a solution `best/` and `summit` copy beside solution.py
requirements: requirements.txt # per-problem venv (default: a shared one with numpy, scipy, pandas, scikit-learn)
unit_tests: # optional correctness gate, frozen at run start
root: tests
command: ["{python}", "-m", "pytest", "-q", "{tests}"]
data_dir: data # default: data/ if the problem has one
allow_internet_during_solution: falseFiles that hillclimb picks up by name when the problem folder has them:
| file | what it is for |
|---|---|
description.md | the task, as the coding agent reads it (required) |
contract.md | the interface solution.py must implement, added to every prompt |
interface.py | the output format as Python objects: described in the prompt, checked by hillclimb verify (Interfaces) |
requirements.txt | the packages the solution and the scorer may import, with requirements: |
plot.py | draws a solution for hillclimb plot (A picture of the solution) |
fingerprint.py | a solution as a vector, for hillclimb similarity |
landscape.py | the terrain hillclimb surface draws a search on |
chart_baselines accepts any number of label: score entries. Each becomes a named horizontal
reference line in hillclimb chart, in the order written, so you can see how far a search is from a known result. A numeric baseline
automatically adds the line named baseline; chart_baselines is only needed for additional
references. A file-based baseline is still evaluated as c000, but cannot become a fixed reference
line until its score is known.