problems / verifier-contracthillclimb

3. Verifier contract

Exit 0 means valid; the score is a file, not a line of stdout.

The verifier is the scoring process the engine starts: verifier.sh, or whatever verifier: in problem.yaml names, and it must be executable (chmod +x). It drives solution.py itself — run it, import it, shell out to it — and reports the score. A verify.py beside it, as in the catalog problems, is only how that verifier happens to score; hillclimb never runs it directly:

problems/my-problem/verifier.sh
#!/usr/bin/env bash
set -euo pipefail

"$HILLCLIMB_PYTHON" "$HILLCLIMB_SOLUTION"   # writes ./submission.csv

rm -f "$HILLCLIMB_RESULT"                   # only the scorer may score
"$HILLCLIMB_PYTHON" problem/verify.py       # writes $HILLCLIMB_RESULT
exit 0the verifier accepted the candidate; declared unit tests must still pass
non-zerothe candidate is buggy and routes to the debug operator
$HILLCLIMB_RESULTthe score: {"score": <float>, "report": {...}, <other numeric keys>}, or a bare number
$HILLCLIMB_PYTHONthe managed runtime venv's interpreter (bare python resolves via PATH: wrong interpreter)
$HILLCLIMB_SOLUTIONthe solution path for this run (replicate-dir aware)
$HILLCLIMB_SPLITvalidation or holdout
$HILLCLIMB_REPLICATE_SEEDset when the engine runs repeated replicates (also exported as the legacy $HILLCLIMB_TRIAL_SEED)

The result file is both the score carrier and the completion proof: the engine deletes it before every run, so a stale file can never fake success, and exit 0 without one is a contract violation rather than a silent zero. Reading the score from a file rather than stdout is what keeps it honest — agent-authored code runs inside the verifier and shares its stdout.

The command runs with cwd = the candidate's working dir (solution.py, plus ./problem/ and ./data/ symlinks). Validation runs get a credential-scrubbed environment; --holdout runs in a directory agents never see, with the full environment (private holdout data may be gated).

Every run is confined by the sandbox: it writes only to its candidate's folder, and it has no network unless the problem sets allow_internet_during_solution: true.

Private data: a scorer of its own

A verifier runs solution.py inside its own process tree, so anything the verifier can read, the solution can read too. Hidden labels, holdout instances or a secret seed next to the verifier are not hidden. To keep data from the solution, give the problem a scorer instead of a verifier, and name what only the scorer may read:

problems/my-problem/problem.yaml
score: verify.py        # the scorer, instead of verifier.sh
private: [hidden]       # files or folders only the scorer reads
time_limit_s: 30        # optional: the engine stops a solution that runs longer
holdout: true           # the scorer is then also run with --holdout

hillclimb then runs two processes, each in a sandbox of its own:

  1. The solution, $HILLCLIMB_PYTHON solution.py, in the candidate's working dir. It cannot read the private: paths, and it gets no credentials, not even on the holdout split. Anything it leaves running is stopped when it exits, and any result file it wrote is deleted.
  2. The scorer, verify.py (or an argv list), in the same dir. It reads what the solution wrote and the private data, and writes the score to $HILLCLIMB_RESULT, exactly as a verifier would. On the holdout split it is passed --holdout.

Coding agents cannot read the private: paths either, and they are left out of the file list in the agent's prompt. A private: path may sit inside the problem folder (hidden) or anywhere else (an absolute path). Declaring private: without score: is an error. hillclimb.Problem(score=...) problems always run in two steps, and take private=[...] for the same purpose.

The problem's folder and its data are read-only to agents and solutions in every problem, so a solution can never rewrite the scorer that judges it.

Two more keys cover the other shapes:

  • run: run.py replaces the plain solution.py step with the problem's own runner. It runs in the solution's sandbox, so it can import the solution and call a function it defines (select_items(items, capacity)), and writes what it got back for the scorer. The bundled knapsack problem works this way.
  • holdout_inputs: [holdout] are inputs of the hidden split, such as the holdout instances or their seed. The solution and the scorer read them only when they run on the holdout split. Coding agents never read them, and neither does a validation run, whose output the agents see.
uv run hillclimb verify my-problem --repeat 5   # score it outside a search

hillclimb verify is the fastest way to check a new verifier, and --repeat reports the spread between identical runs — an improvement smaller than that is noise, not progress. The bundled knapsack and circle-packing problems (hillclimb problem get <problem>) are the two reference shapes: a runner that calls the solution's function plus a scorer with holdout inputs, and a verifier that runs the solution then scores its output.

Make it hard to fool

Whatever the verifier fails to check, the search will eventually exploit. Score invalid outputs as zero, score a solver rather than a fixed answer, and keep some instances hidden with holdout: true and a scorer of its own.