Problem

A Problem is the thing you hand a climber, the way you hand a dataset to a model. It can be a problem that exists already, or one you define right there in Python.

One that exists

from hillclimb import Problem

problem = Problem("fitness-landscape")      # a pre-created problem, or one in your problems/ folder
problem = Problem("problems/my-problem")    # a folder with a problem.yaml
problem = Problem("mlebench://spaceship-titanic")   # a benchmark provider's problem

The name is looked up in the hillclimb dir's problems/ folder. If it is not there but hillclimb ships it, it is copied in the first time a climber uses it, which is what hillclimb problem get does by hand. problem.spec() gives you what the engine loads: the description, the metric, the verifier command and so on. problem.description is the text a coding agent gets.

One of your own

A problem is a scoring function. The coding agent writes a solution.py, hillclimb runs it, then calls your function in the directory it ran in. Whatever number your function returns is the score.

from pathlib import Path
from hillclimb import Problem


def closeness_to_pi(run_dir: Path) -> float:
    guess = float((run_dir / "answer.txt").read_text())
    return -abs(guess - 3.14159265358979)


problem = Problem(
    "closest-to-pi",
    score=closeness_to_pi,
    higher_is_better=True,
    description="Write your best approximation of pi to answer.txt.",
    output="answer.txt",              # the file a solution has to write
    baseline=-3.14159265358979,       # the score of a trivial answer, as the floor
)

argument	what it does
`name`	the problem's id. Also the name of its folder under `problems/`
`score`	your scoring function. It gets the directory the solution ran in, and returns a number, or a dict with a `score` and other numbers to record alongside
`higher_is_better`	which way is up. `True` by default
`metric`	what the score is called in charts and prompts. `"score"` by default
`description`	what you tell the coding agent. Say what the solution should do and what to write
`output`	the file a valid solution writes. `submission.csv` by default
`baseline`	a score to use as the floor, so the first candidate has something to beat
`files`	extra files to put in the problem's folder, as `{name: text or path}`. A sample, some data, a helper module
`requirements`	pip packages the solution may use. Without it the solution runs in hillclimb's shared runtime
`time_limit_s`	how long one run of the solution may take; a run that takes longer is a failed attempt
`private`	files or folders only `score` may read, such as hidden labels. Coding agents and solutions cannot open them
`chart_baselines`	named scores drawn as lines in `hillclimb chart`

The first time a climber searches it, hillclimb writes the problem's folder under problems/: a problem.yaml, the description, a verifier, and a small verify.py that imports your function and calls it. From then on it is a problem like any other. hillclimb verify closest-to-pi scores its baseline, hillclimb run closest-to-pi runs it from the CLI, and hillclimb watch shows it. problem.save() writes the folder without searching anything.

A couple of things to keep in mind:

score has to be a plain function at the top level of a .py file, because the verifier imports that file in a separate process. A lambda, a method, or a function defined inside another one will not work, and Problem(...) tells you so right away.
The verifier imports your whole file, so keep the code that starts a search under if __name__ == "__main__":.
If your function raises, the candidate is marked buggy and the traceback is kept with it. The climber then treats it like any other failed attempt.

Problem

One that exists

One of your own

On this page