#5244 [Engine] serve: accept Codex multi_agent_v2 agent_message and multi_agent_call(_output) items on /v1/responses
closed · @bsorensen110 · 0 comments · View on GitHub
BenchmarksServer & APIAMD / HIPModels & quantsSecurityDocumentationWindowsLinux
Description
## Problem and result
Codex CLI `0.121.0+` with `features.multi_agent_v2` enabled (and the OpenAI Responses multi-agent API) sends inter-agent hops as `agent_message` items on `POST /v1/responses` (`codex-rs/core/src/tasks/regular.rs`, `codex-rs/protocol/src/models.rs`), splitting each hop into an `input_text` envelope header (`Message Type: ... / Task name: ... / Sender: ... / Payload:\n`) and an `encrypted_content` payload part, alongside optional `multi_agent_call` and `multi_agent_call_output` replay items (`developers.openai.com/api/docs/guides/responses-multi-agent`).
On `main` (`ffa80d731`), `serve/responses.py:input_messages()` only handles `message`, `reasoning`, `function_call`, `function_call_output`, `custom_tool_call`, `custom_tool_call_output`, `additional_tools`, and `item_reference`. Any `agent_message` or `multi_agent_call(_output)` item falls through to the unsupported-type branch and fails the request with HTTP 400:
```json
{"error": {"message": "input items of type 'agent_message' are not supported", "type": "invalid_request_error", "param": "input[6].type", "code": "unsupported_parameter"}}
```
Additionally, when a subagent is spawned with the default `fork_turns="all"` (full-history fork), Codex's `collab_tools_enabled()` (`codex-rs/core/src/tools/spec_plan.rs`) includes the `collaboration` tool namespace (`spawn_agent`, `wait_agent`, `followup_task`, `send_message`, `interrupt_agent`, `list_agents`) on `/root` (`get_agent_path().is_none() == true`), but omits `collaboration` from `tools` on spawned subagents (`get_agent_path().is_some()`) whenever a custom model's metadata does not set `multi_agent_version = "v2"`. Because the chat template renders `<tools>` near the beginning of the system message (~token 1,500, before the first conversation-cache checkpoint at token ~13,517), dropping `collaboration` from `tools` shifts the entire remaining ~53,000 chars of inherited prompt and causes a **0-token KV prefix cache miss** (`0 reused + 12,974 read in 13,037 ms`) on the first turn of every spawned subagent.
Related upstream work: open draft PR #4210 (`Anemone95:responses-agent-message`, `a00a4be7c`) also addresses `agent_message` items by rewriting `encrypted_content` parts into `input_text` and passing the list to `_content()`. This branch (`33155a053` + `245f20314`, split into two atomic commits on `ffa80d731`) is offered as a fuller alternative (or can be folded into #4210) that additionally:
1. Preserves the item's `author` and `recipient` (or `sender` / `agent.agent_name` fallback per the Responses API reference) in a leading `[agent_message from <author> to <recipient>]` routing line, and joins parts without doubling trailing newlines after `Payload:\n`.
2. Decrypts our own `strata.r1:` `encrypted_content` boxes when `box["text"]` is a `str` (matching `reasoning` item handling in `input_messages()`), passes plaintext or foreign string tokens through verbatim, and degrades damaged/non-string `encrypted_content`, non-string `text` parts, or unknown part types to `[encrypted payload unavailable]` / `[payload part not rendered]` without failing the whole multi-agent turn.
3. Updates `_order_tool_results()` to scan back to the nearest preceding `assistant` message when an `agent_message` (which is a `user` message) arrives between parallel tool calls and their `function_call_output` items, so tool results are still sorted by the assistant turn's `call_id` order.
4. Accepts `multi_agent_call` and `multi_agent_call_output` replay items (`action` / `name` tool call + `call_id`-paired tool output).
5. Preserves the identified parent conversation's (`(session_id, parent_thread_id)`) omitted namespace tool definitions in `prompt_tools()` (`_fork_prompt_tools`) for **prompt rendering only** (keeping `request_tools()`'s callable parser tools and `names` map restricted to the child request's declared tools, and leaving requests unchanged when parent ownership is not established) — turning a first-turn `fork_turns="all"` subagent KV prefix cache miss (`0 reused + 12,974 read in 13.04s`) into a **99.8% prefix cache hit (`13,517 reused + 808 read in 1.78s`, saving 11.25s of prefill per spawned subagent)**.
## Approach
- **Commit 1 (`33155a053` — `serve: accept Codex multi_agent_v2 agent_message and multi_agent_call(_output) items on /v1/responses`)**:
- Add `_agent_message_parts(item, author, recipient)` in `serve/responses.py`: prepends `[agent_message from {author} to {recipient}]`, accepts `content` as a `list` of parts, a plain `str`, or `None`, renders `input_text` / `output_text` / `text` when `text` is `str` or `None` (degrading non-string `text` to `[payload part not rendered]`), `input_image` as `[image]`, and `encrypted_content` by decoding valid `strata.r1:` boxes whose `text` is a `str`, passing non-prefixed strings through, and degrading non-string or damaged boxes to `[encrypted payload unavailable]`, joining without doubling trailing newlines.
- Handle `kind == "agent_message"` in `input_messages()`, closing `open_turn` and clearing `thinking`, and update `_order_tool_results()` to locate the nearest preceding `assistant` turn (`next((messages[k] for k in range(i - 1, -1, -1) if messages[k]["role"] == "assistant"), {})`).
- Extend `function_call(_output)` / `custom_tool_call(_output)` branches in `input_messages()` to include `multi_agent_call` (reading `action` when `name` is absent) and `multi_agent_call_output`, and update `docs/DETAILS.md`.
- **Commit 2 (`245f20314` — `serve: preserve parent namespace tools on fork_turns=all subagent turns for KV prefix reuse`)**:
- Record each conversation's rendered prompt tools in `_fork_prompt_tools[(session_id, thread_id)]` (capped at 16 entries) inside `prompt_made()`.
- When a `multi_agent_v2` turn (`agent_message` in `input`) identifies a recorded parent conversation via `(session_id, parent_thread_id)` in `x-codex-turn-metadata` (`_codex_parent()`) and carries an inherited `function_call` in a `namespace` omitted from `tools`, `prompt_tools()` splices that parent conversation's `namespace.*` tool definitions into the returned `shown` prompt tool list after their recorded predecessor, while keeping `request_tools()` (`tools` and `names`) restricted to the request's declared tools.
## Validation
**Passed:**
- Before (`origin/main` at `ffa80d731`): `input_messages({"input": [AM_TASK]})` raised `ResponsesError("input items of type 'agent_message' are not supported", "input[0].type")` (RED).
- After (`33155a053` + `245f20314`, rebased on `origin/main` `ffa80d731` / Strata `0.2.10` on Linux x86_64):
```bash
PYTHONPATH=. python3 -m unittest serve.test_responses serve.test_frontend serve.test_server -v
```
- `serve.test_responses`: **49 tests, OK (1 skipped)** — includes 6 new tests in `MultiAgentV2` (`test_agent_message_item_parses`, `test_agent_message_encrypted_ciphertext_degrades`, `test_agent_message_prompt_golden`, `test_agent_message_between_turns_keeps_tool_pairing`, `test_multi_agent_call_and_output_items_parse`, `test_fork_turns_subagent_restores_namespace_tools_for_prefix_cache`).
- Combined `serve.test_responses` + `serve.test_frontend` + `serve.test_server`: **434 tests, OK (1 skipped)**.
- Live Codex CLI `0.166.0` (`features.multi_agent_v2 = true`) end-to-end runs on `qwen3.8-flash-next-iq3_s` (`gfx1201`, AMD Radeon AI PRO R9700, 64 GB RAM, Linux, `--batch 2` + `STRATA_BATCH_DF2=1`):
- Single & multi-turn subagents (`spawn_agent` -> `wait_agent` -> `followup_task` -> `wait_agent`) and 2-way parallel subagents (`fork_turns="none"` + `fork_turns="all"`, `send_message`, `list_agents`, `wait_agent`): all 25 `/v1/responses` HTTP requests returned `200 OK` (`status: "completed"`).
- First-turn `fork_turns="all"` subagent prefill A/B (`codex exec` spawning `/root/arith` after `/root` Turn 1 parked `13,522` prompt + `86` generated tokens):
- Without parent namespace restoration (`33155a053`): `strata serve: prompt 12974 tokens = 0 reused + 12974 read in 13037 ms (995.2 tok/s)` (**0 tokens reused**, **13.04s** prefill).
- With parent namespace restoration (`245f20314`): `strata batch: slot 1 takes 14325 tokens (copied in 3.8 ms)` / `strata serve: prompt 14325 tokens = 13517 reused + 808 read in 1783 ms (453.1 tok/s)` (**13,517 tokens reused in 3.8 ms**, **1.78s** prefill — **7.3x faster**, saving **11.25s** per spawned subagent).
**Failed:**
- None observed.
**Not run:**
- Windows and macOS test suites (change is pure Python in `serve/`).
## Limits
- Strata does not execute subagents itself; client-side orchestrators (such as Codex CLI) execute `spawn_agent` / `send_message` / `followup_task` / `wait_agent` tool calls and replay the resulting `agent_message` and `multi_agent_call(_output)` items on subsequent `/v1/responses` turns.
- Foreign `encrypted_content` ciphertexts encrypted with another provider's key cannot be decrypted locally and are rendered as their opaque string token.
- `fork_turns="none"` subagents intentionally start without inherited parent turns (so they do not carry a historical `collaboration.*` `function_call` in `input`); to keep `collaboration` in `tools` on `fork_turns="none"` subagents as well, set `"multi_agent_version": "v2"` on the model entry in Codex's `model_catalog_json`.
Related on this site
Quick links to related pages.