#392 Report: Flash-Next IQ3_S on 2x RTX 5060 Ti (PCIe gen3) as a coding agent in Forge

本文

This is a field report, not a bug. Strata runs Qwen3.8-Flash-Next IQ3_S on two 16 GB consumer cards as the backend of a real coding agent. **Every code run below went through Forge**, a VS Code extension, talking to `serve/server.py` as an openai-compatible provider: multi-round tool calls (read/edit/search files, run builds) with contexts growing past 86K tokens.

**Hardware:** i9-9900KF (water-cooled), 128 GB DDR4-3000, 2x RTX 5060 Ti 16 GB plus an RTX 3060 (not used by Strata). PCIe gen3 x8/x8/x4, Windows 10. Prebuilt engine 0.1.30.

**Numbers** (streamed, server timings, `--spec 4`, MTP draft):

| | 1 GPU | 2 GPUs (layer split auto) |
|---|---|---|
| decode, short prompt | 38.7-41.5 tok/s | 50.5-60.5 tok/s |
| decode after a 23.8K prompt | 33.6-39.8 tok/s | 46.3-58.2 tok/s |
| cold prefill, 23.8K tokens | 860 tok/s (27.7 s) | ~996 tok/s (23.9 s) |

`tools/calibrate.py` on 2 GPUs found `--pcie-frac 0.20 --spec-min-p 0.70` at 52.6 tok/s, against 50.4 tok/s for the defaults. That gain is within run-to-run noise on this box. Moving from an 8700K to a 9900KF raised 2-GPU decode by about 10%.

For comparison, on the same machine llama.cpp's best Flash-Next setup (3 GPUs plus CPU MoE offload) does about 9.5-12 tok/s decode, bound by RAM bandwidth. Strata is roughly 4-5x faster here.

**Agent results** (2 GPUs, three Forge sessions on 2026-09-30, all on the Forge repo itself):

| Task | Time | Tool calls | Failed | Outcome |
|---|---|---|---|---|
| Read-only review of the repo, then fix what it found | 99 min | 214 | 5 | Found 3 real bugs (all verified), fixed them with tests; committed (15 files) |
| Write an implementation plan for a feature | 17 min | 96 | 0 | Plan written; passed the repo's plan-format check |
| Revise the plan after review, then implement it | 69 min | 176 | 9 | Implemented and committed (21 files) |

- Totals: about 3 hours, 486 tool calls, 14 failed (~3%). Failures are counted from the start of each tool result, so treat it as a lower bound.
- Of the 14:
  - 10 were ordinary model mistakes that the tools reported clearly:
    - 6 edits whose quoted text no longer matched the file, including one whitespace slip.
    - `ls` with no shell, a guessed file path, two options that can't be combined, and `npx --noEmit` with `tsc` missing (it timed out).
  - 3 were partly Forge's fault, all fixed since. Each started with a malformed call, but the client handled it badly:
    - It crashed with an unhelpful error when the required `command` was missing (twice).
    - It accepted `"True"` as a string for a boolean option, so the command ran in the foreground and timed out.
  - 1 was a misnamed argument (`new_text` for `new_str`), reported cleanly.
  - Malformed arguments (missing, misnamed or wrong type) were 4 of the 14.
- Context ran past 86K tokens. Two sessions were compacted mid-task and carried on afterwards. Decode stayed around 47 tok/s past 75K context.
- Caveats: a separate reviewing agent (Claude) checked and finished both changes before they were committed, so this is not unassisted work. The model also over-explores: it reads and searches more than it needs before acting.
- One thing on Strata's side: tool calls are not constrained to the tool's JSON schema. 4 of the 14 failures were calls that a schema check would have rejected: a required argument missing twice, an argument with the wrong name, and a boolean sent as the string `"True"`. Optional schema-constrained tool calls would remove that class of failure.

<img width="1677" height="1510" alt="Image" src="https://github.com/user-attachments/assets/c2d7dd5e-0ba5-460e-96af-b4a43ff6e2e9" />

: an agent turn mid-run — tool calls in progress, model name visible]

<img width="1723" height="1534" alt="Image" src="https://github.com/user-attachments/assets/9d41e409-e8aa-4060-aec5-6ee20c77c5d8" />

: Strata's server log during that turn — tok/s and the per-request prompt read lines]

**What works well:**
- Tool calling over the openai-compatible API handled long multi-round agent sessions, including parallel tool calls.
- The 2-card split worked with no manual layer tuning.

**What hurts for agent use:** the fixed per-request prompt cost (2-3 s for a ~130-token tool result) is paid every round. I've put the details and a trace on #340, not in a separate issue.