Code Exec Metric

Isolating a candidate means running it in a fresh Python interpreter, one per candidate, so whatever it does happens in a process the host can kill. Inline exec has no such boundary. A generated while True: pass blocks the eval forever, and a generated import shutil; shutil.rmtree('/') does exactly what it says. The runner embeds the candidate into a script with repr(), runs that script in a fresh python -I -c interpreter, reads the assertion results back from stdout, and kills the process if it overruns. The host eval process keeps running.

HumanEval, MBPP, BigCodeBench, and LiveCodeBench all use a subprocess sandbox, and some layer Docker on top. I stop at the subprocess because it is portable, it is stdlib, and it catches the failure modes that matter for an educational eval. Production deployments add seccomp, network isolation, and a read-only filesystem. That hardening work lives outside this track.

A code_exec task is one whose targets are assertion strings instead of reference text. The runner extracts a fenced code block from the generation, builds a test harness around it, and runs the result. extract_code returns the contents of the first fenced block. If the text has no fences at all it returns the whole text, and if a fence is opened but never closed it returns None, which scores 0.0 without running anything.

The score is a fraction in [0, 1]: assertions passed over assertions supplied. A task with three assertions where two pass scores 0.667. The runner returns the same shape whatever fails. Subprocess crashes map to a normalised error code, so a Python traceback does not bubble up to the harness.

The denylist is a set of modules candidate code is not allowed to import. It is import-based: before running the candidate, the runner script replaces builtins.__import__ with a guard that raises ImportError("denied module: <name>") for any denied module or submodule of one. The list is deliberately conservative: subprocess, socket, requests, urllib (with its request, error, and parse submodules), ctypes, shutil, http.client, asyncio.subprocess. os.system is not an import, so it is patched separately to raise OSError.

I don't pretend this is bulletproof. Determined adversarial code can escape any in-process sandbox in Python, so the denylist is a backstop. The wall-clock timeout and the output cap carry the load.

DENIED = {
    "os.system": True,
    "subprocess": True,
    "socket": True,
    "shutil": True,
    "requests": True,
    "urllib": True,
    "ctypes": True,
}

This dict is illustrative. The real set lives in the runner template in main.py. The guard and the os.system patch run before the candidate is compiled, so a denied import fails when the candidate executes and surfaces as an error exit code, not syntax_error.

The wall-clock timeout is the budget of real elapsed seconds a candidate gets before the runner kills it. The default is three seconds, enforced by a subprocess.Popen plus a select loop that reads stdout against a monotonic deadline. When the deadline passes, the runner kills the process and records a timeout exit reason for the task. The task scores zero and the runner moves on.

The budget is the timeout_s argument to run_candidate and score_code_exec. Long-running unit tests can ask for more, and main.py clamps the value to the range 0.1 to 30 seconds to keep the suite bounded.

The output cap is a hard limit on how much stdout a candidate can produce, set at 256 KB. Without one, a subprocess can flood stdout and exhaust host memory. The runner streams stdout into a buffer and kills the child as soon as the running total crosses the cap. The result is recorded as exit_code = error with the detail string "output overflow". In practice this shows up when a generation accidentally writes an infinite loop that prints.

Pass-at-k is the probability that at least one of k sampled solutions passes. The unbiased estimator used by HumanEval and friends computes it from n independent samples per task, c of which pass. The probability that a sample of size k drawn from the n contains at least one passing solution is:

pass_at_k(n, c, k) = 1 - C(n - c, k) / C(n, k)

When n - c < k the binomial in the numerator is zero, so the value is 1. The implementation handles that edge case directly. If k > n it returns float(c > 0), and invalid arguments (negative counts, k <= 0, or c > n) raise ValueError. pass_at_k(n, c, k) is exposed for the leaderboard layer in lesson 74.

An exit code is a label for how a run ended, recorded as metadata beside the score. The runner returns one of five outcomes per task:

• pass when all assertions passed.
• assertion_fail when the code ran but at least one assertion failed.
• syntax_error when compile() raises a SyntaxError on the candidate.
• timeout when the wall clock expired.
• error for any other crash, including denylist hits (an ImportError raised when the candidate runs), exceptions at exec time, a non-zero subprocess exit, and output overflow (overflow surfaces with detail "output overflow").

The score is still a fraction. Downstream lessons can decide whether a timeout counts as zero or as missing data.

The harness does not give you a real sandbox. It does not run untrusted code from the open web, and it does not handle stateful tasks like file I/O or network calls. Those need a container or a microVM. What you get is the contract: an isolated subprocess, a denylist, a timeout, an output cap, a clean exit-code vocabulary, and pass-at-k math.

main.py defines extract_code, run_candidate, score_code_exec, and pass_at_k. The subprocess runner script is built as a string and passed as -c to a fresh Python interpreter. The tests in code/tests/test_exec.py exercise the exit codes plus pass-at-k against worked examples drawn from the HumanEval style.

Read main.py top to bottom. The runner template is the load-bearing piece. Keep reading the assertion loop until you can predict the JSON envelope it writes back to the parent process.

Once the subprocess shape works, portability is the next problem. The runner reads the child's stdout with os.set_blocking and select, which is POSIX-oriented, so it is not portable as written. The cleanest fix is to put the runner in a Docker image. After that, replace assertion strings with real unit test files so the eval matches what production CI does. At that point, stop calling assertion strings tests. They are toy tests with toy failure modes.