3. Verifier contract
Exit 0 means valid; the score is a file, not a line of stdout.
The verifier is the scoring process the engine starts: verifier.sh, or whatever verifier: in
problem.yaml names, and it must be executable (chmod +x). It drives solution.py itself — run
it, import it, shell out to it — and reports the score. A verify.py beside it, as in the
catalog problems, is only how that verifier happens to score; hillclimb never runs it directly:
#!/usr/bin/env bash
set -euo pipefail
"$HILLCLIMB_PYTHON" "$HILLCLIMB_SOLUTION" # writes ./submission.csv
rm -f "$HILLCLIMB_RESULT" # only the scorer may score
"$HILLCLIMB_PYTHON" problem/verify.py # writes $HILLCLIMB_RESULT| exit 0 | the verifier accepted the candidate; declared unit tests must still pass |
| non-zero | the candidate is buggy and routes to the debug operator |
$HILLCLIMB_RESULT | the score: {"score": <float>, "report": {...}, <other numeric keys>}, or a bare number |
$HILLCLIMB_PYTHON | the managed runtime venv's interpreter (bare python resolves via PATH: wrong interpreter) |
$HILLCLIMB_SOLUTION | the solution path for this run (replicate-dir aware) |
$HILLCLIMB_SPLIT | validation or holdout |
$HILLCLIMB_REPLICATE_SEED | set when the engine runs repeated replicates (also exported as the legacy $HILLCLIMB_TRIAL_SEED) |
The result file is both the score carrier and the completion proof: the engine deletes it before every run, so a stale file can never fake success, and exit 0 without one is a contract violation rather than a silent zero. Reading the score from a file rather than stdout is what keeps it honest — agent-authored code runs inside the verifier and shares its stdout.
The command runs with cwd = the candidate's working dir (solution.py, plus ./problem/ and
./data/ symlinks). Validation runs get a credential-scrubbed environment; --holdout runs in a
directory agents never see, with the full environment (private holdout data may be gated).
Every run is confined by the sandbox: it writes only to its candidate's folder,
and it has no network unless the problem sets allow_internet_during_solution: true.
Private data: a scorer of its own
A verifier runs solution.py inside its own process tree, so anything the verifier can read, the
solution can read too. Hidden labels, holdout instances or a secret seed next to the verifier are
not hidden. To keep data from the solution, give the problem a scorer instead of a verifier, and
name what only the scorer may read:
score: verify.py # the scorer, instead of verifier.sh
private: [hidden] # files or folders only the scorer reads
time_limit_s: 30 # optional: the engine stops a solution that runs longer
holdout: true # the scorer is then also run with --holdouthillclimb then runs two processes, each in a sandbox of its own:
- The solution,
$HILLCLIMB_PYTHON solution.py, in the candidate's working dir. It cannot read theprivate:paths, and it gets no credentials, not even on the holdout split. Anything it leaves running is stopped when it exits, and any result file it wrote is deleted. - The scorer,
verify.py(or an argv list), in the same dir. It reads what the solution wrote and the private data, and writes the score to$HILLCLIMB_RESULT, exactly as a verifier would. On the holdout split it is passed--holdout.
Coding agents cannot read the private: paths either, and they are left out of the file list in
the agent's prompt. A private: path may sit inside the problem folder (hidden) or anywhere else
(an absolute path). Declaring private: without score: is an error. hillclimb.Problem(score=...)
problems always run in two steps, and take private=[...] for the same purpose.
The problem's folder and its data are read-only to agents and solutions in every problem, so a solution can never rewrite the scorer that judges it.
Two more keys cover the other shapes:
run: run.pyreplaces the plainsolution.pystep with the problem's own runner. It runs in the solution's sandbox, so it can import the solution and call a function it defines (select_items(items, capacity)), and writes what it got back for the scorer. The bundledknapsackproblem works this way.holdout_inputs: [holdout]are inputs of the hidden split, such as the holdout instances or their seed. The solution and the scorer read them only when they run on the holdout split. Coding agents never read them, and neither does a validation run, whose output the agents see.
uv run hillclimb verify my-problem --repeat 5 # score it outside a searchhillclimb verify is the fastest way to check a new verifier, and --repeat reports the spread
between identical runs — an improvement smaller than that is noise, not progress.
The bundled knapsack and circle-packing problems (hillclimb problem get <problem>) are the
two reference shapes: a runner that calls the solution's function plus a scorer with holdout
inputs, and a verifier that runs the solution then scores its output.
Make it hard to fool
Whatever the verifier fails to check, the search will eventually exploit. Score invalid outputs as
zero, score a solver rather than a fixed answer, and keep some instances hidden with
holdout: true and a scorer of its own.