#3025 Tensor parallel (--tp 2) on 2× RTX 3080: patch attached, correct but not usable yet (~2 % of the layer split's prefill) — open for pickup

Description

> **Status: not usable yet.** `--tp 2` is correct on every path that works, but it runs at **~2 % of the layer
> split's prompt throughput**, and its fast path (860–990 tok/s) produces `non-finite logits` from the first QSA
> layer on. **Do not run this in production.**
>
> This is an **open handover, not a submission**: the perf breakdown and the correctness bug are the useful part,
> and what is left is written down below. It is **not** a request to the maintainers — #156 was closed as *not
> planned*, and these numbers support that decision. It is a starting point for anyone who wants to continue it.

**Patch:** branch `tp2-wip` — https://github.com/ZeaMQ/Strata/tree/tp2-wip (head `e771f5cb`, on top of `296b59d2`, engine 0.1.44.1)
**Write-up:** `docs/TP2-HANDOVER.md` (English) / `docs/TP2-HANDOVER.zh-CN.md` (中文)

**If you want to pick it up, read [The three steps I would try next](#the-three-steps-i-would-try-next) first.**

---

# TP=2 (tensor parallel) is correct but slower than the layer split; its fast path is broken — handing over



I spent a long session adding `--tp 2` to a CUDA build of 0.1.44.1 (`296b59d2`) on 2x RTX 3080 20 GB and I am

posting this as a **half-finished** attempt rather than a PR, because the honest result is: it works, and it is

slower than what you already have.



## Relation to #156



This follows up **#156** ("support tensor parallelism for multi gpu setup"), closed 2026-09-30 as *not planned

for now: decode is limited by the experts, so tensor parallelism of the dense part gains little here*. I built

it to get the measurement that thread did not have, and **the numbers support that decision**: the correct TP

path lands at about 2 % of the layer split's prefill throughput (37–52 vs 2 114.8 tok/s), with host grouping

at 47.8 % of the TP prefill and the TP reduce at another 16.2 %. So please read this as evidence for closing

it, plus the two real bugs found on the way — not as a request to reopen it.



## What happened



- `--tp 2` with the prompt read through the two-rank verify windows is **correct at every length I tried**

  (108 / 3 458 / 6 856 / 14 060 prompt tokens, all read in full, no non-finite logits), and VRAM lands at

  **19 909 / 19 867 MiB**, the same as the layer split on this pack.

- But prompt throughput on that correct path is **37–52 tok/s**, against **2 114.8 tok/s** for the layer split

  on the same machine, same pack, same day. Decode 43–58 tok/s against 99.3.

- The fast path (one batched prefill, `STRATA_TP_FASTPROMPT=1`) does reach **860–990 tok/s** but produces

  `non-finite logits (#879)` from the first QSA layer on, for any prompt larger than one KV page. So it ships

  disabled, with the windows as the TP default.



## Strata version, GPU, OS



Strata **0.1.44.1**, base commit **`296b59d2`** (2026-10-14) · 2x RTX 3080 20 GB (PCIe 4.0 x8) + Quadro P1000

for vision · Windows 11 (10.0.26200) · Ryzen 9 5900X, 80 GiB · CUDA source build.



**Provenance:** AI-assisted, not human-reviewed; the fast path below survived thirteen fix attempts unsolved.

This is a bug report with a patch attached, not a submission.



## Engine log



```

strata serve: non-finite logits (#879) in a prompt window at position 6851 (row 0): its tokens are not emitted

strata serve: prompt 6856 tokens = 0 reused + 6851 of 6856 read in 7972 ms (859.4 tok/s), 0 generated

strata dbg gdn: layer 3 qsa attn       583872 of 635904 non-finite (max |x| 2.288e+06)

strata dbg gdn: layer 3 qsa bo(reduce) 527360 of 529920 non-finite

strata dbg gdn: layer 3 staged=1 kv_mode=1 page_size=4 s.n_head_kv=2

```



## Where I got to



- Queries are fine (`qsa q(in)` clean); the attention output is not, so the QSA layer reads wrong KV cells.

- The GDN block's own counts and state offsets are correct; the MoE non-finite is downstream.

- `page_size` is 4 (not 64) and `staged=1` — the prompt attention reads the staged pool with `m.ident_table`.

  I had assumed 64 for most of the session, which invalidated a dozen hypotheses.

- Page-stride changes in both directions were tried and reverted: writer `page*2` / reader `page*1` gives

  92 % non-finite, writer `page*1` / reader `page*1` gives 100 % - so the write side is not the lever.

- Two real defects were found and fixed on the way: the QSA query stride (packed at `qh*256`, read at the

  model's 6144) and the expert-cache sizing (fixed 3/4 shrink threw away 1.83 GiB; that fix alone is what

  puts the card at 19.44 GB).

- `STRATA_GDN_KEYHEAD=0` is currently **required** in TP: `gdn_rec_kh_kernel` is not parameterised and still

  uses the model's full HV/HK.



## The three steps I would try next



1. Print `kv_append_kernel`'s `row_id` for the first 8 staged cells and `prompt_attn_kernel`'s `r` for the same

   cells, and diff them.

2. Dump `m.ident_table` (`prefill.cpp:1204`) and confirm `ident[i] == i`.

3. Re-derive with `page_size == 4`.



## What I did not test



CUDA only — no HIP or SYCL build (AGENTS.md asks for all three). One machine, one pack, one model. No

50K–100K measurement, no recall/needle suite, no bitwise single-card regression against `HEAD`, no soak.



## Licence and prior art



Strata is MIT (c) 2026 Niko1221 and the Strata contributors; this report asserts no warranty and claims no

fitness. Model weights / native pack / expert profile are separate artifacts and are not included. No

credentials are included.



This work's tensor-parallel rules were written against **llama.cpp / ggml**, which Strata already carries as

a pinned, gitignored dependency (`third_party/`, `.gitignore:31`). Both are verbatim **MIT**, copyright

(c) 2023-2026 The ggml authors, and neither is redistributed with the source. Strata cites them inline with

`file:line` — `shard.hpp` cites `handle_mul_mat:600-603`, `GGML_BACKEND_SPLIT_AXIS_PARTIAL:591-593` and the

qwen4exp shapes at `qwen4exp.cpp:242-247`. If review finds a function that is an adaptation rather than a

reimplementation, the ggml notice goes beside it.