problems / defining-problemshillclimb

1. Problem structure

A problem is a folder, and a problem is its verifier.

A problem is a folder, and a problem is its verifier. Users define new problems without changing Python code:

problems/my-problem/
├── problem.yaml
├── description.md
├── verifier.sh            # the contract: exit 0 = valid, write the score
└── data/                  # optional runtime inputs

A problem scores its solutions in one of two shapes:

  • A verifier, verifier.sh, that runs the solution and then its scorer. hillclimb runs only verifier.sh; a verify.py is just the scorer it calls. It is the simplest to read, and 2. A simple verifier is a whole problem in this shape.
  • Two steps, score: in problem.yaml (and run: to call a function the solution defines). hillclimb runs the solution and the scorer in a sandbox each, so the scorer can keep data the solution never sees, and no bash is needed, on Windows too. hillclimb problem new and the bundled knapsack problem are in this shape; see a scorer of its own.

Minimum problem.yaml:

problems/my-problem/problem.yaml
metric: my-score
higher_is_better: true             # true or false, unquoted

Everything else is optional:

problem_id: my-problem           # default: the folder's name
description: description.md      # the default
verifier: verifier.sh            # the default
score: score.py                  # instead of a verifier: the scorer of a two-step problem
run: run.py                      # with score: the problem's own runner for the solution
time_limit_s: 60                 # with score: a solution that runs longer is a failed attempt
private: [hidden]                # with score: files only the scorer may read
holdout: true                    # engine also runs `verifier.sh --holdout`
holdout_inputs: [holdout]        # with score: inputs only the holdout split reads
contract: contract.md            # what solution.py must be/do (prompt section)
baseline: 0.5                    # scored at t=0 as the floor candidate; numeric values also become chart lines
                                 # or a file: `baseline: baseline.py`, scored at t=0 as c000
baseline_files:                  # or files the floor is scored on as they are
  submission.csv: sample_submission.csv
chart_baselines:                 # optional named horizontal lines in `hillclimb chart`
  previous best: 0.73
output_artifacts: [submission.csv]  # files of a solution `best/` and `summit` copy beside solution.py
requirements: requirements.txt   # per-problem venv (default: a shared one with numpy, scipy, pandas, scikit-learn)
unit_tests:                      # optional correctness gate, frozen at run start
  root: tests
  command: ["{python}", "-m", "pytest", "-q", "{tests}"]
data_dir: data                   # default: data/ if the problem has one
allow_internet_during_solution: false

Files that hillclimb picks up by name when the problem folder has them:

filewhat it is for
description.mdthe task, as the coding agent reads it (required)
contract.mdthe interface solution.py must implement, added to every prompt
interface.pythe output format as Python objects: described in the prompt, checked by hillclimb verify (Interfaces)
requirements.txtthe packages the solution and the scorer may import, with requirements:
plot.pydraws a solution for hillclimb plot (A picture of the solution)
fingerprint.pya solution as a vector, for hillclimb similarity
landscape.pythe terrain hillclimb surface draws a search on

chart_baselines accepts any number of label: score entries. Each becomes a named horizontal reference line in hillclimb chart, in the order written, so you can see how far a search is from a known result. A numeric baseline automatically adds the line named baseline; chart_baselines is only needed for additional references. A file-based baseline is still evaluated as c000, but cannot become a fixed reference line until its score is known.