BuildPython

Parallel Tool Calls and Streaming with Tools

Three independent weather lookups run one after another cost three round trips. Run them in parallel and the total time collapses to the slowest single call. The major providers now emit multiple tool calls in a single turn, and the payoff is real, but the plumbing is subtle. This lesson covers both halves: the parallel fan-out and the reassembly of streamed arguments, with the id-correlation trap front and center.

TIME: ~75 minutesPREREQUISITES: Phase 13 · 02 (function calling deep dive)

Learning Objectives

Explain why `parallel_tool_calls: true` exists and when to disable it.
Correlate streamed argument chunks to the right tool-call id during parallel fan-out.
Reassemble partial `arguments` strings into complete JSON without parsing early.
Run a three-city weather benchmark that demonstrates sequential vs parallel latency.

The Problem

Without parallel calls, an agent answering "what is the weather in Bengaluru, Tokyo, and Zurich" does this:

user -> LLM
LLM -> call get_weather(Bengaluru)
host -> run executor, reply with result
LLM -> call get_weather(Tokyo)
host -> run executor, reply with result
LLM -> call get_weather(Zurich)
host -> run executor, reply with result
LLM -> final text answer

Three LLM round trips, each of which also pays the executor latency. That is far from the ideal wall-clock time, because each round trip pays model latency on top of the executor's.

With parallel calls:

user -> LLM
LLM -> call get_weather(Bengaluru); call get_weather(Tokyo); call get_weather(Zurich)
host -> run all three executors concurrently, reply with three results
LLM -> final text answer

One LLM round trip. Executor time is the maximum of the three, not the sum. Production benchmarks on OpenAI, Anthropic, and Gemini show 60 to 70 percent wall-clock reduction on fan-out workloads.

The price is correlation. When the three calls complete out of order, your results must carry the matching tool_call_id so the model can line them up. When results stream, you must assemble partial argument fragments into complete JSON before executing. Gemini 3 added unique ids in part to solve a real-world issue where two parallel calls to the same tool were indistinguishable.

Warm-up0/3

1. A model can emit only one tool call per turn. How many call-then-reply cycles does it need to fetch weather for three cities before it writes its final answer?Warm-up

2. A host runs json.loads on the string {"city": "Beng. What happens?Warm-up

A short Python snippet that sets s to the cut-off string {"city": "Beng and calls json.loads on it, above a dashed box asking what happens.

3. A script fires 50 requests at once at a web API that allows about 5 requests per second. What most likely happens?Warm-up

Check: The Problem0/2

1. What are parallel tool calls?Define

2. Why did Gemini 3 add unique ids to function calls?Explain

A function call record with the fields name get_weather and args, plus a dashed id field marked as added in Gemini 3, beside a dashed box asking why the id was added.

The Concept

Enabling parallel

OpenAI. parallel_tool_calls: true on by default. Set false to force serial.
Anthropic. Parallel via disable_parallel_tool_use: false (default on Claude 3.5 and up). Set true for serial.
Gemini. Parallel-capable; tool_config.function_calling_config.mode = "AUTO" lets the model decide.

Disable parallel when tools have ordering dependencies (create_file then write_file), when one call's output informs another's input, or when the rate limiter cannot handle fan-out.

Id correlation

Each call the model emits has an id, and each result the host returns must include the same id. Without it, results are ambiguous.

OpenAI. tool_call_id on each tool-role message.
Anthropic. tool_use_id on each tool_result block.
Gemini. id on each functionResponse (Gemini 3 and up; Gemini 2 matched by name which broke for same-name parallel calls).

Running calls concurrently

The host runs each call's executor on its own thread, coroutine, or remote worker. The simplest harness uses a thread pool; production uses asyncio with asyncio.gather or structured concurrency. Order of completion is unpredictable, so the id is the identifier.

One common bug: reply with results in call-list order instead of completion order. This usually works because the model only cares about tool_call_id, but if a result is dropped or duplicated, out-of-order submission makes debugging harder. Prefer to reply in completion order with explicit ids.

Streaming tool calls

When the model streams, arguments arrive in pieces. Three separate streams of chunks for three parallel calls interleave on the wire. You need one accumulator per id.

Shape by provider:

OpenAI. Each chunk is choices[0].delta.tool_calls[i].function.arguments (partial string). The chunk carries index (position in the call list). You accumulate per-index, read id when it first appears, and parse JSON when finish_reason = "tool_calls".
Anthropic. Stream events are message_start, then one content_block_start per block with type tool_use (containing id, name, empty input). content_block_delta events carry input_json_delta chunks. content_block_stop closes each block.
Gemini. streamFunctionCallArguments (Gemini 3 and up) emits chunks with a functionCallId so calls interleave cleanly. Before Gemini 3, streaming returned one complete call at a time.

Partial JSON and the parse-early trap

You cannot parse arguments until it is complete. Partial JSON such as {"city": "Beng is not valid and will raise. The correct gate is the provider's end-of-call signal: OpenAI's finish_reason = "tool_calls", Anthropic's content_block_stop, or Gemini's stream-end event. Only then attempt json.loads. A more robust approach uses an incremental JSON parser that yields events as structure completes; OpenAI's streaming guide recommends this for UX that shows a live "thinking" indicator. Brace-counting is unreliable as a completeness test (braces inside quoted strings or escaped content cause false positives) and should only be used as an informal debug heuristic.

Out-of-order completion

call_A: fast API, returns first
call_B: slow API, returns second
call_C: median API, returns third

The host reply must still cite the ids:

[{role: "tool", tool_call_id: "call_A", content: ...},
 {role: "tool", tool_call_id: "call_B", content: ...},
 {role: "tool", tool_call_id: "call_C", content: ...}]

Order in the reply does not matter for correctness on OpenAI or Anthropic. Gemini accepts any order so long as ids match.

Benchmark: sequential vs parallel

The harness in code/main.py simulates three executors with 400, 600, and 800 ms latency. Sequential runs it in 1800 ms total. Parallel runs it in max(400, 600, 800) = 800 ms. The gap is the sum minus the max, so it widens as you add tools.

Real-world caveat: parallel calls stress downstream APIs. A 10-way fan-out to a rate-limited service will fail. Phase 13 · 17 covers gateway-level backpressure; retry semantics are planned for a future phase.

Streaming fan-out wall-clock

If the model itself streams, you can start executing as soon as one call's arguments are complete, rather than waiting for all calls to finalize. This is an optimization OpenAI documents but not all SDKs expose. The harness here does it: as soon as the simulated stream yields a complete argument object, the host kicks off that call.

Check: The Concept0/26

1. In basic function calling, the model emits a tool call and the host replies with a result. What ties that result back to the right call when several calls are outstanding?Define

Three boxes labelled tool call, each waiting for a result, and one result box on the right with dashed lines to all three and a dashed box asking what ties them together.

2. What does the parallel_tool_calls flag control?Define

3. What does setting Anthropic's disable_parallel_tool_use to true do?Define

4. What is an accumulator in streamed tool calls?Define

$Two stream event rows for call_B carrying the argument chunks '{"city' and '":"Tokyo"}', an arrow into a dashed box labelled accumulator with a question mark.$

5. What is out-of-order completion?Define

6. What does a dependency graph describe in a tool registry?Define

A tool registry panel with three tool boxes named search_flights, book_flight and get_weather, and a dashed box asking for the registry's dependency graph.

7. What is the parse-early trap?Define

Four event rows for call_B from a toy stream: call_start, two args_delta chunks and call_stop, beside a json.loads(buf) box and a dashed box asking what the parse-early trap is.

8. What does Gemini's streamFunctionCallArguments provide?Define

A box naming Gemini 3 and up and the feature streamFunctionCallArguments, above a dashed box asking what each streamed event carries.

9. What is a completion-order reply?Define

10. With parallel tool calls enabled, what determines the total executor wall-clock time for three fanned-out calls?Explain

11. Which field does each provider require the result message to echo back to correlate it with its originating tool call?Explain

Three columns for OpenAI, Anthropic and Gemini 3 and up, each showing a result message type (tool-role message, tool_result block, functionResponse) above a dashed box asking for the echoed field.

12. Which combination of default parallel-call behavior and flags is correct?Explain

Three provider boxes: OpenAI with parallel_tool_calls, Anthropic with disable_parallel_tool_use and Gemini with no setting named, above a dashed box asking for the default behavior and flags of each.

13. Why does the host need one buffer per call id when arguments stream in?Explain

Three buffers labelled buf[call_A], buf[call_B] and buf[call_C], each empty, under a note that arguments stream in, and a dashed box asking why there is one buffer per id.

14. Why does Anthropic's guidance point to disabling parallelism for consequential mutations on the same resource?Explain

15. Why does the lesson list a rate limiter as a reason to disable parallel calls?Explain

A host box sending tool calls through a rate limiter box to a service box, with a dashed box asking why the lesson lists this as a reason to disable parallel calls.

16. What is the correct signal to gate parsing a streamed tool call's arguments as JSON?Explain

A list of streamed events: message starts, tool call starts, three arguments chunks and an ellipsis, above a dashed box asking when the buffer is safe to pass to json.loads.

17. Why is brace-counting an unreliable test for whether streamed JSON is complete?Explain

18. How does the wire format for streamed tool-call arguments differ across providers?Explain

Three columns headed OpenAI, Anthropic and Gemini 3, each holding a dashed box asking what a streamed chunk carries.

19. The host replies to out-of-order parallel calls in call-list order rather than completion order. What does the lesson say about this?Explain

Two rows of boxes: results finishing in the order call_A, call_C, call_B, and the host's reply in call-list order call_A, call_B, call_C, with a note that each reply message carries its tool_call_id. — Finish order is illustrative.

20. A host exposes create_file and write_file, and the model emits both in one turn for the same path. What should the host do?Apply

An assistant message box holding create_file(path=P) and write_file(path=P) for the same path, above a dashed box asking what the host should do.

21. A host calls json.loads on each streamed buffer after every chunk and sees intermittent decode errors. Which ingredient should have stopped this?Analyze

A typed loop that appends each streamed chunk to buf and calls json.loads on buf every time, a note about intermittent decode errors, and a dashed box asking for the missing ingredient.

22. A developer reuses a StreamAccumulator keyed on event id with real OpenAI chunks and gets a KeyError on later chunks of a call. What is the likeliest cause?Analyze

The toy accumulator's lookup lines idx = event.get('id') and buf = self.buffers[idx], a box listing the OpenAI chunk path choices[0].delta.tool_calls[...].function.arguments, a note about a KeyError on later chunks, and a dashed box asking for the likeliest cause.

23. After switching a 10-call batch from sequential to parallel, some calls to one rate-limited service start failing, though sequential never did. Which cause fits?Analyze

Two boxes, sequential with 10 calls one after another and no failures, and parallel with 10 calls where some fail, both pointing at one rate-limited service box, with a dashed box asking for the likeliest cause.

24. Design A waits until the whole stream ends and then runs every call. Design B submits each call at its own stop event. What does B gain over A?Analyze

Three horizontal bars for call_A, call_B and call_C streaming, each ending in a red call_stop dot, with a dashed line where the stream ends and two boxes describing Design A, which waits until the whole stream ends and then runs every call, and Design B, which submits each call at its own stop event.

25. A registry holds read_weather and read_stock (independent reads from separate APIs) plus create_ticket and add_comment (which needs the new ticket's id). Which scheduling fits?Design

A registry of four tools: read_weather and read_stock described as independent reads from separate APIs, create_ticket, and add_comment, which needs the new ticket's id, above a dashed box asking for the scheduling.

26. A UI should show tool-call arguments live while they stream, yet executors must only run on valid input. Which design fits?Design

A UI panel that shows tool-call arguments live with the partial text '{"city":"Zu' and an executor box that must only run on valid input, above a dashed box asking for a design that satisfies both.

Use It

code/main.py has two halves. The first runs three simulated weather calls sequentially and in parallel using concurrent.futures.ThreadPoolExecutor and prints wall-clock time. The second half replays a fake streaming response, with chunks of arguments for three parallel calls interleaved on one stream, and reassembles them per-id with StreamAccumulator. No LLM, no network, just the reassembly logic.

What to look at:

The sequential timer hits 1.8 seconds. The parallel timer hits 0.8 seconds on the same fake latencies.
Chunks from different calls interleave on the stream. The accumulator buffers them per id and parses a call's JSON only after that call's stop event.
The executor kicks off as soon as an id's arguments finalize, not after all streams end.

Check: Use It0/5

1. In the Use It benchmark the simulated latencies are 400, 600 and 800 ms. How long does the parallel run take?Apply

Three boxes for Bengaluru at 400 ms, Tokyo at 600 ms and Zurich at 800 ms, above a dashed box asking for the total time of the parallel run.

2. In main.py, Tokyo's simulated latency is changed to 1000 ms and the other two stay at 400 and 800. What are the theoretical sequential and parallel times?Apply

A typed edit of the SIMULATED_LATENCY_MS dictionary with Bengaluru 400, Tokyo 1000 and Zurich 800, above two dashed boxes asking for the theoretical sequential and parallel times.

3. In main.py's fake stream, call_A receives the chunks '{"city"', ':"Beng' and 'aluru"}' before its call_stop. Which call prints its 'args complete' line first, and with what arguments?Apply

Four event rows for call_A from main.py's fake stream: three args_delta chunks and a call_stop, a note that other calls' events are interleaved, the print statement that reports a completed call, and a dashed box asking for the first line printed. — Only call_A's events are drawn; the rest of the stream is left out here.

4. In main.py, call_C has received only its first chunk, '{"city":"Zu', and no call_stop yet. What does try_parse() return for it?Apply

The fields of a CallBuffer for call_C: id call_C, name get_weather, args_buf '{"city":"Zu' and done False, with a call to buf.try_parse() and a dashed box asking what it returns.

5. run_parallel in main.py is changed to use max_workers=1. Roughly how long does it take for the three cities?Apply

The simulated latency dictionary with Bengaluru 400, Tokyo 600 and Zurich 800 and a ThreadPoolExecutor with max_workers=1 mapping executor_weather over the cities, above a dashed box asking for the elapsed time.

Ship It

This lesson produces outputs/skill-parallel-call-safety-check.md. Given a tool registry, the skill audits which tools are safe to parallelize, which have ordering dependencies, and which would overwhelm downstream rate limits, and returns a revised registry with per-tool parallel_safe flags.

Check: Ship It0/1

1. What does the parallel-call-safety-check skill return when given a tool registry?Explain

Author's Note

I'd run main.py and read the second demo's output before the first one's. The args complete lines print in the order the stop events arrive, and that is the moment each call is submitted to the pool, so you can see execution start before the stream has finished. The thread pool timing in demo 1 is the easy part. The accumulator is where the real work is.

My bet is that the id is the whole trick. StreamAccumulator keeps one CallBuffer per id and only parses after call_stop sets done. Nothing else in it is clever. Interleaving stops being a problem once every chunk says which call it belongs to.

I'd also keep the toy in proportion. The fake stream tags events with an id, while the OpenAI shape described above tags chunks with an index, so a real OpenAI accumulator keys on position and reads the id when it first appears. The temperatures in the results come from hash(city), which changes between runs, so only the timings and the id-to-result mapping mean anything. And the results dict is filled in submission order, not completion order, which is fine here because each result stays attached to its id.

Exercises

Run code/main.py and vary the simulated latencies. Confirm that the parallel-to-sequential ratio is approximately max/sum (real runs deviate slightly from the ideal because of thread scheduling, serialization, and harness overhead). At what latency distribution does parallel stop mattering?
Extend the accumulator to handle a "call was cancelled mid-stream" case by dropping its buffer and emitting a cancelled event. What provider documents this case explicitly? Check Anthropic's content_block_stop semantics and OpenAI's finish_reason: "length" behavior.
Replace the thread pool with asyncio.gather. Benchmark both. You should see small wins on async because of lower context-switch cost, but only if executors do real I/O.
Pick two tools that should NOT parallelize (e.g. create_file then write_file). Add an ordering_dependency graph to the registry and gate the parallel fan-out on that graph. This is the minimum machinery for dependency-aware scheduling, which a future agent-engineering phase formalizes.
Read OpenAI's parallel-function-calling section and Anthropic's disable_parallel_tool_use docs. Identify the one real-world tool type where Anthropic recommends disabling parallelism. (Hint: consequential mutations on the same resource.)

Key Terms

Term	What people say	What it actually means
Parallel tool calls	"Fan-out in one turn"	Model emits multiple tool calls in a single assistant message
`parallel_tool_calls`	"OpenAI's flag"	Enable or disable multi-call emission
`disable_parallel_tool_use`	"Anthropic's inverse"	Opt-out flag; default is parallel enabled
Tool call id	"Correlation handle"	Per-call identifier the result message must echo
Accumulator	"Stream buffer"	Per-id string buffer for partial `arguments` chunks
Out-of-order completion	"Fastest first"	Parallel calls finish in unpredictable order; ids are the glue
Dependency graph	"Ordering constraints"	Tools whose outputs feed into inputs of other tools; cannot parallelize
Parse-early trap	"JSON.parse exploded"	Attempting to parse an incomplete `arguments` string
`streamFunctionCallArguments`	"Gemini 3 feature"	Streamed argument chunks with unique id per call
Completion-order reply	"Don't wait for all"	Reply with results as they arrive, keyed by id