Function Call Dispatcher

The dispatcher is the layer that takes one tool call from the harness loop and turns it into either a result or a JSON-RPC-shaped error envelope. It sits between the harness loop (lesson twenty) and the tool registry (lesson twenty-one). The transport (lesson twenty-two) feeds the loop, the loop hands a tool call to the dispatcher, and the dispatcher looks up the handler in the registry and runs it.

Timers, retries, and idempotency live here and nowhere else. The loop does not know about them, the registry does not, and the handler does not. That isolation is the point.

A timeout is a hard ceiling on how long one handler attempt may run. Each tool has a default, carried as timeout_ms on its registry record, and the harness can pass a per-call override that wins over the default. The dispatcher enforces it with asyncio.wait_for. When the timer fires, the handler task is cancelled and the dispatcher returns DispatchError(kind="timeout").

Whether a timeout retries depends on the tool. A db.write that timed out may or may not have committed, so retrying it can duplicate the write. The dispatcher reads the idempotent flag from the registry record: idempotent tools retry, non-idempotent tools do not.

Exponential backoff with jitter means each retry waits longer than the last, with a random stretch added so that many callers do not retry in lockstep. The policy here is three attempts maximum.

attempt 1  -> delay 0
attempt 2  -> delay 0.1s * (1 + random[0..0.5])
attempt 3  -> delay 0.4s * (1 + random[0..0.5])

Only timeout and transient errors retry. A schema error, a not_found, or an internal error never does. Schema errors are deterministic, so a retry cannot change the outcome and only burns the budget.

The dispatcher also respects the harness budget, but it checks it before the retry loop starts. If the caller has zero tool calls remaining, dispatch() returns kind="budget_exceeded" with attempts=0 and no handler attempt runs.

An idempotency key is a caller-supplied string that identifies one logical operation, so the dispatcher can tell a retry of that operation apart from a new call. It exists for a production bug that is easy to miss: a retry fires while the original is still in flight. The first call hangs at four point nine seconds, just under the timeout, and the retry fires at five seconds. Now two requests race against the same backend, and if the tool is payments.charge, you charged twice.

The dispatcher accepts an optional idempotency_key. If a call arrives while the same key is in flight, the dispatcher waits on the in-flight future and returns its result. Completed keys stay in the cache for sixty seconds to absorb late retries.

The key is the caller's job. The harness derives it from the planner: f"{step_id}:{tool_name}:{hash(args)}". The dispatcher does not invent keys, because a key derived from arguments alone makes two semantically different calls look the same.

The error envelope is the single shape every failed dispatch returns, whatever went wrong underneath.

DispatchError
  kind        : "timeout" | "transient" | "schema" | "not_found" | "internal" | "budget_exceeded"
  message     : str
  attempts    : int
  jsonrpc_code: int   (one of -32601, -32602, -32603)

The harness loop maps kind to its next state. schema and not_found go to on_error and trigger a replan. timeout and transient also go to on_error, and whether they replan depends on how many attempts were used. budget_exceeded triggers on_budget_exceeded.

A concurrency limit caps how many dispatches run at the same moment. Without one, gather(*calls) starts every coroutine at once. Forty tool calls means forty open sockets or forty subprocess pipes, and most backends do not like forty parallel connections from one client.

The dispatcher wraps gather in a semaphore with a default limit of eight. Each call acquires the semaphore before dispatching and releases it on completion. The caller still gets gather-shaped output, but the scheduling underneath is bounded.

code/main.py defines Dispatcher, DispatchError, and TransientError. The dispatcher takes a registry on construction, and the async dispatch(name, args, ...) is its only entry point. Per-attempt timeouts are applied inline inside _run_with_retries using asyncio.wait_for. gather_bounded(calls) runs many dispatches under the concurrency limit.

code/tests/test_dispatcher.py covers timeout firing, retry on transient, no retry on schema error, idempotency dedupe (two concurrent calls with the same key collapse to one handler invocation), and concurrency limiting (the semaphore in action).

The tests use asyncio.sleep(0) and deterministic Counter-based handlers, so they finish in milliseconds and do not depend on wall-clock timing.

Production dispatchers usually add two things on top of this one. The first is structured logging at every transition. The loop's event stream already gives you some of that, but the dispatcher should also emit dispatch.attempt and dispatch.retry events. The second is a circuit breaker: after N failures in a window, a tool gets a cool-down period during which dispatches return immediately with a new kind="circuit_open" (not one of the six kinds above) instead of attempting the handler. Both fit on top of this dispatcher without changing the contract.

Lesson twenty-four glues the dispatcher to a plan-and-execute agent, so you see all four pieces in motion.