2. A simple verifier

Here is a complete problem: get as many digits of pi right as you can. The coding agent writes solution.py, which computes pi and writes it to answer.txt. The verifier checks how many digits are right.

problems/closest-to-pi/
├── description.md
├── problem.yaml
├── verifier.sh
└── verify.py

The task, as the coding agent reads it:

description.md

# Closest to pi

Write `solution.py`. It must compute pi as accurately as it can, without
typing the digits in or importing them, and write the number to `answer.txt`.

The name of the score, and which way is better:

problem.yaml

metric: digits-correct
higher_is_better: true
baseline: 0

The verifier. This is the file hillclimb runs; it runs the solution, then the scorer:

verifier.sh

#!/usr/bin/env bash
set -euo pipefail

"$HILLCLIMB_PYTHON" "$HILLCLIMB_SOLUTION"   # the solution writes answer.txt
rm -f "$HILLCLIMB_RESULT"                   # only the scorer may write a score
"$HILLCLIMB_PYTHON" problem/verify.py       # the scorer writes it

The scorer. hillclimb never runs it directly: verifier.sh calls it. It compares the answer with the real digits and writes the score to the file hillclimb gives it:

verify.py

import json
import os

PI = "3.14159265358979323846264338327950288419716939937510"

answer = open("answer.txt").read().strip()
correct = 0
for ours, true in zip(answer, PI):
    if ours != true:
        break
    correct += 1

with open(os.environ["HILLCLIMB_RESULT"], "w") as f:
    json.dump({"score": max(correct - 2, 0)}, f)   # digits after "3."

That is all. Make the verifier executable, then try it on a solution before you start a search:

chmod +x problems/closest-to-pi/verifier.sh
echo 'open("answer.txt", "w").write(str(22 / 7))' > guess.py
hillclimb verify closest-to-pi --solution guess.py

  run 0: digits-correct = 2

22 / 7 gets two digits right. A search will try to do better, and the verifier is what decides whether it did. 3. Verifier contract has the rules this verifier follows, and what else it can report.