#6423 [Engine] 0.2.17 sm_120 defaults regress prefill and cache-reuse TTFT on a mixed-arch layer split with the 12.0 card first

open · @wronglebowsk · 2 comentários · No GitHub

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Descrição

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

Manual clone built with CUDA 13.4 (arch 75;86;120), served under systemd.

### What happened

Updating a tri-GPU layer-split install from v0.2.16 (engine 0.1.62) to v0.2.17 (engine 0.1.63), the new default-on sm_120 package regressed prefill and time-to-first-token. The release's Blackwell numbers are single-card (RTX 5090 / 5060 Ti); on a mixed-arch split where the compute-capability-12.0 card is **CUDA0, the first pipeline stage**, the package is a net loss, and `STRATA_SM120_PERF=0` recovers all of it.

- Problem: with the 0.2.17 defaults, prefill fell 5–7% at every prompt size (0.2k–33k tokens), conversation-cache reuse TTFT rose from 0.21 s to 0.48 s, and a 69-token prompt's TTFT rose from 0.57 s to 1.35 s.
- Expected: the release's "faster on NVIDIA Blackwell cards" gains, or at least no regression on a layer split.
- Actual: the table below. Setting `STRATA_SM120_PERF=0` (with `STRATA_ROUTE_TAIL_SKIP=7` to keep the pre-0.2.17 decode behavior) restores parity with 0.1.62 across the 3.7k–33k band and the reuse TTFT. Quality is unchanged throughout: a paired 180-item battery (MMLU/GSM8K/ARC-Easy, temp 0, thinking-on) gave 0 discordant pairs between 0.1.62 and every 0.1.63 arm (McNemar p=1.0).

The engine numbers the cards differently from nvidia-smi: on this box CUDA0 is the 5060 Ti, so the short-read / HC up-mix / decode-switch changes act on the stage that ingests every prompt first. The package seems to assume the 12.0 card is the whole pipeline (or at least not the first stage of a mixed-arch split).

A second, smaller finding is residual and **not attributed**: with `SM120_PERF=0`, `ROUTE_TAIL_SKIP=7` and `ADAPT_FETCH=0` all set, tiny prompts still pay +0.1–0.6 s TTFT vs 0.1.62 (table below). It persists across every knob combination we tried, so it looks like a general 0.2.17 change rather than one of the named defaults.

### Smallest steps or command that reproduce it

1. Build v0.2.17 and serve an IQ3_XXS Flash-Next pack on a mixed-arch split with the 12.0 card first: `--layer-split 18,33` on RTX 2080 Ti + RTX 5060 Ti + RTX 3080 Ti (engine order: CUDA0 = 5060 Ti), `--pipeline-windows 2`, fused prompt kernels (`STRATA_PF_FUSED=1`, `STRATA_PF_FUSED_SM75=1`).
2. Measure prefill tok/s and TTFT at 3.7k / 15k / 30k tokens and the conversation-cache reuse TTFT, with the 0.2.17 defaults.
3. Set `STRATA_SM120_PERF=0` (and `STRATA_ROUTE_TAIL_SKIP=7` for the decode comparison), restart, re-measure: the prefill and reuse-TTFT regressions disappear.

### Reproduction status

Reproduced locally. Five arms (0.1.62 reference; 0.2.17 defaults; +tail-skip 7; +SM120_PERF=0; +ADAPT_FETCH=0), two passes each, identical randomized prompts, ten-minute settles, medians. The battery ran on the reference, the defaults arm, and the final pinned arm.

### Strata version

Strata 0.2.17 · engine 0.1.63. Measured on our fork's merge of upstream v0.2.17 (merge commit 734d16700); the engine-side local deltas are #6084 (draft-head VRAM accounting), #6107 (serve body timeout), the #6149 line (identical to upstream's 928e31305), and the CPU pool's NUMA `slice_ms_` vector — none touches the sm_120 prompt/decode paths, which are toggled by upstream's own `STRATA_SM120_PERF`. Reference: v0.2.16 · engine 0.1.62.

### Graphics card, memory and system

RTX 2080 Ti 11 GB (sm_75) + RTX 5060 Ti 16 GB (sm_120, CUDA0) + RTX 3080 Ti 12 GB (sm_86), PCIe, no P2P; engine PCIe probes 6.7 / 6.1 / 11.8 GB/s · 188 GB RAM · Ubuntu 26.04.1 LTS, kernel 7.3.4 · CUDA 13.4.

### Model, size and settings

Qwen3.8-Flash-Next UD-IQ3_XXS · `--max-context 524288` · `--layer-split 18,33` · `--pipeline-windows 2` · `--kv k8v4` · `--ple-io ram` · vision on · thinking-on serving (temp 0 for measurements).

### Logs (sanitized)

```
strata generate: GPU 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0
strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split 18,33)
strata: prompt kernels: chunks of 1024 tokens or more run 48 of 48 layers on the fused kernels; shorter chunks run 48 on the staged fused entry.
```

### Measurements

Medians of two passes, tok/s and seconds. "Pinned" = `SM120_PERF=0` + `ROUTE_TAIL_SKIP=7` + `ADAPT_FETCH=0`.

| case | metric | 0.1.62 | 0.2.17 defaults | pinned | defaults vs 0.1.62 | pinned vs 0.1.62 |
|---|---|---|---|---|---|---|
| prefill 3.7k | tok/s | 543 | 518 | 536 | −4.6% | −1.3% |
| prefill 15k | tok/s | 1,343 | 1,265 | 1,325 | −5.8% | −1.3% |
| prefill 30k | tok/s | 2,058 | 1,942 | 2,068 | −5.6% | +0.5% |
| cache 33k cold | tok/s | 2,182 | 2,041 | 2,174 | −6.5% | −0.4% |
| cache 33k reuse | TTFT s | 0.21 | 0.48 | 0.21 | +128% | −1% |
| 69-token prompt | TTFT s | 0.57 | 1.35 | 0.65 | +137% | +14% |
| decode 512 | tok/s | 105.8 | 99.1 | 101.7 | −6.4% | −3.9% |
| decode 1024 | tok/s | 102.7 | 96.5 | 104.7 | −6.1% | +1.9% |
| turn 2 (reuse) | decode tok/s | 131.2 | 111.6 | 141.0 | −15% | +7.5% |
| turn 3 (reuse) | decode tok/s | 117.5 | 122.9 | 129.3 | +4.6% | +10.1% |

The decode rows mix two effects: tail-skip-off (recoverable with `ROUTE_TAIL_SKIP=7`, which alone restores decode 512/1024 to parity) and the sm_120 package. `STRATA_POOL_DEP` is inert here (the NUMA pool split gates it), and the CPU prefill share did not arm on this split.

Residual with all three pins, vs 0.1.62 — the unattributed finding:

| case | metric | 0.1.62 | pinned | delta |
|---|---|---|---|---|
| prefill 0.2k (~228 tokens) | TTFT s | 1.71 | 2.27 | +33% |
| decode 128 | TTFT s | 1.34 | 1.52 | +13% |
| decode 512 | TTFT s | 1.11 | 1.19 | +7% |
| structured JSON (136 tokens) | TTFT s | 3.95 | 4.20 | +6% |

Ruled out for the residual: `SM120_PERF`, `ROUTE_TAIL_SKIP`, `ADAPT_FETCH` (each toggled in isolation), `POOL_DEP` (inert under the NUMA split). Not yet tried: `STRATA_PCIE_START=old`, `STRATA_PF_HC_REGION=0`.

### Suggested fixes (scope for agreement)

1. Gate the sm_120 package on topology (e.g. arm it only when the 12.0 card is not the first stage of a mixed-arch split), or document the layer-split caveat beside the release's single-card numbers. The start-up line could name the state per stage, as the adaptive-fetch line does.
2. The short-prompt residual may be worth a look independently; we can run targeted arms (`PCIE_START=old`, `HC_REGION=0`) and attach per-request data if useful.

Per-arm JSONL (every request, both passes) and the battery item set are available on request.

No site

Links install, modelos, releases.