Defining problems

A problem is a folder, and a problem is its verifier. Users define new problems without changing Python code:

problems/my-problem/
├── problem.yaml
├── description.md
├── verifier.sh            # the contract: exit 0 = valid, write the score
└── data/                  # optional runtime inputs

Minimum problem.yaml:

problems/my-problem/problem.yaml

problem_id: my-problem
metric: my-score
higher_is_better: true
description: description.md
output_artifacts: [submission.csv]  # use [submission.json] for JSON-native tasks
time_budget_s: 900

Everything else is optional:

verifier: verifier.sh            # the default
holdout: true                    # engine also runs `verifier.sh --holdout`
contract: contract.md            # what solution.py must be/do (prompt section)
baseline: 0.5                    # scored at t=0 as the floor candidate; numeric values also become chart lines
chart_baselines:                 # optional named horizontal lines in `hillclimb chart`
  previous best: 0.73
requirements: requirements.txt   # per-problem venv (default: shared csv venv)
unit_tests:                      # optional correctness gate, frozen at run start
  root: tests
  command: ["{python}", "-m", "pytest", "-q", "{tests}"]
data_dir: data
allow_network: false

chart_baselines accepts any number of label: score entries. Each becomes a named horizontal reference line, in the order written. A numeric baseline automatically adds the line named baseline; chart_baselines is only needed for additional references. A file-based baseline is still evaluated as c000, but cannot become a fixed reference line until its score is known.

Bundled problems

These problems are small, local, and require no download, so they are good for exercising parallel searches in hillclimb watch:

problem	objective
`circle-packing`	maximize total radius for 26 circles in a unit square
`heilbronn-11`	maximize the smallest triangle area among 11 points
`tsp-200`	minimize a 200-city Euclidean TSP tour
`labs-60`	minimize length-60 binary autocorrelation energy

Start the suite, then open the TUI:

uv run hillclimb run problems/demo-suite.yaml --name "Optimization demo" --budget 10m
uv run hillclimb watch

submission.csv

the 50-circle packing one solution produced, drawn in the unit square

The verifier contract

What the scoring process has to do, and the environment it gets.

Optional features

Unit tests, metrics, reports, tunable parameters, per-instance scores.

Bundled problems

The verifier contract

Optional features

On this page