Execution sandboxes

How Hillclimb Uses Sandboxes

hillclimb starts two kinds of process that run code nobody has read:

The agents, which work with their permission prompts off.
The verifier, which runs the solution.py an agent wrote.

Both run on your machine, as you.

So both run inside a sandbox. It is on by default, it is the operating system's own, and it needs no container.

operating system	sandbox
macOS	`sandbox-exec` (Seatbelt), built into macOS
Linux	bubblewrap
Windows	through WSL2: bubblewrap, as on Linux. Natively, none: `hillclimb` runs unsandboxed and says so

See it for yourself

hillclimb sandbox check runs a script that behaves like a hostile solution, inside the same sandbox a search uses, and lists what happened to every attempt:

hillclimb sandbox check

$ hillclimb sandbox check A hostile script, run inside the sandbox (sandbox-exec)     files✓   write to its own candidate folder                     allowed✓   write to your home folder                             blocked✓   write to the hillclimb dir                            blocked    secrets✓   read the search's journal (it holds holdout scores)   blocked✓   read ~/.ssh                                           blocked✓   read Claude Code's login and history (~/.claude)      blocked    network✓   connect to the internet (1.1.1.1:443)                 blocked✓   look up a name (example.com)                          blocked    processes✓   signal a process outside the sandbox                  blocked    agent without internet✓   reach a host that is not allowed (example.org)        blocked✓   connect past the proxy                                blocked✓   reach an allowed host (example.com)                   allowed   The sandbox holds: 10 attempts blocked, and its own folder stays writable.

It exits with code 1 if one attempt got through. Whatever the script managed to write outside its folder is removed again. hillclimb connect shows in one line whether the sandbox starts on your machine.

What a sandboxed process can do

	allowed	blocked
write	its candidate's folder, the temp dirs, the caches of `huggingface`, `torch` and `matplotlib`	everything else: your home folder, your repositories, the search's journal and `best/`, the Python environments solutions run in, and always the problem's folder and data
read	most of the disk	`~/.ssh`, `~/.aws`, `~/.gnupg`, cloud and GitHub logins, browser profiles, shell history, `.env` files, the other agents' logins, the search's journal and its holdout runs, the sockets of Docker and other container runtimes, and the problem's `private:` paths
network, agents	everything; with `allow_internet_for_agents: false` only their model provider	the rest
network, verifier	nothing; with the problem's `allow_internet_during_solution: true` everything	the rest, name lookups included
processes	its own children	signalling anything outside

An agent also writes to its own operator home under ~/.cache/hillclimb/: Claude Code's and codex's, each with a login of its own, and pi's isolated home and session folder. Your own ~/.claude, ~/.claude.json, ~/.codex and ~/.agents are unreadable to everything a search starts, so your settings, plugins, skills and MCP servers never reach a search, and nothing a search does reaches them.

How it is enforced

The engine stays outside. It is the only process that writes the journal and best/. Every agent call and every verifier run is started inside a sandbox of its own, and everything that process starts stays inside with it:

your machine
│
├── engine (hillclimb run)                 outside: writes the journal and best/
│   └── proxy                              the one way out for agents without internet
│
├── sandbox ── agent (claude-code, pi)     writes its candidate's folder
│               └── every tool and script it starts
│
└── sandbox ── verifier.sh                 writes its candidate's folder, no network
                └── solution.py

hillclimb does not implement the confinement. It writes down the rules and hands them to the mechanism the operating system already has:

OS	what enforces it	install
macOS	`sandbox-exec` (Seatbelt), the mechanism Chrome and Claude Code use	nothing, it is part of macOS
Linux	bubblewrap, the mechanism Flatpak uses	`sudo apt install bubblewrap` (`dnf`, `pacman` alike)
Windows	bubblewrap inside WSL2	in WSL2's Ubuntu: `sudo apt install bubblewrap`. Natively none exists: `hillclimb` runs unsandboxed and says so at every start, pointing here

The rules are readable. On macOS they are one Seatbelt profile per process, on Linux one list of bubblewrap arguments, both written by harness/sandbox.py.

Per agent

agent	how it is confined
`claude-code`	the whole process runs inside the sandbox
`pi`	the whole process runs inside the sandbox
`codex`	by its own sandbox (`--sandbox workspace-write`): writes stay in the candidate's folder and its commands have no network. macOS allows no sandbox inside a sandbox, so `hillclimb` does not put a second one round it, and codex can still read files the table above lists as blocked
`dummy`	runs no model and no tools

Agents without internet

By default an agent may use the internet while it works. To take it away:

hillclimb.yaml

allow_internet_for_agents: false

The agents then reach their model provider and nothing else. Everything they and their tools send goes through a small proxy inside the engine, which lets HTTPS through to the provider's hosts and refuses the rest. The engine's log names every host it refused. Claude Code's web search, web fetch and MCP servers are off, codex's web search is off, and the draft prompt no longer asks for research on the web.

This needs the sandbox. With sandbox: off, or on Windows, a search with claude-code or pi does not start.

Windows, via WSL2

Native Windows has no sandbox hillclimb can use. WSL2 runs a real Linux kernel, so the Linux sandbox (bubblewrap) works there unchanged: run hillclimb inside WSL2's Ubuntu, and agents and solutions are confined exactly as on Linux. (Claude Code takes the same route: its own sandbox supports macOS, Linux and WSL2.)

Once, in PowerShell as administrator, then restart:

wsl --install -d Ubuntu

Then, in the Ubuntu shell:

sudo apt update && sudo apt install -y bubblewrap python3-venv git
python3 -m venv ~/hillclimb-env && source ~/hillclimb-env/bin/activate
pip install hillclimb
hillclimb sandbox check        # "The sandbox holds" means agents and solutions are confined

Keep your hillclimb projects in the Linux file system (under ~/), not on the Windows drive (/mnt/c/…): it is much faster there, and it is the setup the sandbox is checked in.

Check it on your machine

scripts/wsl2-check.sh checks the whole sandbox in about two minutes, with the dummy agent (no login, no tokens): hillclimb sandbox check, a problem with hidden holdout data scored on both splits, and a hostile solution in a real search that tries to read the holdout data, change the scorer, write to your home folder and reach the internet. It runs once with the project under ~/ and once on the Windows drive, and ends with a summary to send back:

git clone https://github.com/rebase-energy/hillclimb
pip install ./hillclimb          # the version in the clone (or: pip install hillclimb)
bash hillclimb/scripts/wsl2-check.sh

a good run ends like this (your versions will differ)

=================== summary (send this back)
6.6.87.2-microsoft-standard-WSL2 | Ubuntu 24.04.3 LTS | bubblewrap 0.9.0 | hillclimb 0.7.0
PASS  project in the Linux filesystem
PASS  project on the Windows drive

A FAIL line comes with the output above it that shows what got through.

When it does not start

A search does not start when the sandbox is on and cannot be started. The message says why:

what it says	what to do
bubblewrap is not installed	install it with your package manager
it does not start, on Ubuntu 24.04 or later	Ubuntu keeps the kernel feature bubblewrap uses behind AppArmor: `sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0`
it does not start, inside a container	the container has to allow user namespaces. With Docker: `--security-opt seccomp=unconfined --security-opt apparmor=unconfined --security-opt systempaths=unconfined` (the last one, or bubblewrap cannot mount `/proc`). If you cannot change that, the container is the sandbox: set `sandbox: off`
it does not start, on macOS	`hillclimb` was started inside another sandbox, a coding agent's for example. Run it from a plain terminal

Settings

hillclimb.yaml

sandbox: off              # run everything unsandboxed

hillclimb.yaml

sandbox:                  # or adjust it
  write: [scratch]        # more writable paths, relative to the hillclimb dir
  deny_read: [~/private]  # more unreadable paths
  allow_hosts: [bedrock-runtime.eu-north-1.amazonaws.com]   # more hosts for agents without internet
  local_ports: [8000]     # localhost ports left open when the network is off

HILLCLIMB_SANDBOX=off in the environment switches it off for one command.

What it does not protect

It is a sandbox, not a virtual machine

It stops writes outside the candidate's folder, reads of your keys and traffic to hosts you did not allow. It does not protect against a flaw in the operating system's kernel.

These stay open:

An agent can read its own login, because it needs it.
A sandboxed process can still read most of your files. Add what is private to sandbox.deny_read.
An agent without internet still talks to its model provider, so what it can read can reach that provider.
codex is confined by its own sandbox, which does not hide your keys from it.
On native Windows nothing is confined: run hillclimb inside WSL2.

Execution sandboxes

On this page