#6462 [Benchmark] RTX 4090 (sm_89): sm_86 bit-exact set +6%, sm_120 switches +4% on UD-Q4_K_XL, CPU share 32/6144 cuts 3K-token follow-up turns 26%

open · @ThatsR4d · 0 comentários · No GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationLinux

Descrição

### What did you measure, or what went wrong

Single RTX 4090 (sm_89, 24 GB, PCIe 4.0 x16, `--pcie-mode dma --pcie-frac 0`), i9-14900K (8P+16E), 128 GB DDR5-5200, Ubuntu 26.04 (Linux 7.0), driver 615. Qwen3.8-Flash-Next, DFlash2, 262K context, one request at a time. Items marked 0.1.63 that touch UD quants or turns ran on 0.1.63 built locally from v0.2.17 with the #6339 patch and `--lookup-max-t 16` (otherwise identical; decode matched the released engine within 1%). sm_89 gets no row in `nv_defaults`, so none of the measured-win sets apply by default on a 4090. Decode = mean of 6 x 512-token answers (story / code / reason), interleaved pairs; drift = `tools/quant_quality.py` (teacher-forced).

**1. The sm_86 bit-exact set on sm_89: +6%, bit-identical** (engine 0.1.60, UD-Q4_K_XL)
`STRATA_ADAPT_NOWAIT=1 STRATA_ADAPT_LAG=2 STRATA_SIDE_MIXER=2 STRATA_ATTN_DAILY=1 STRATA_GR_UP75=1 STRATA_GU_SWIGLU=1 STRATA_QSA_ROWS=1 STRATA_B6_MMVQ_ROWS=1`: 74.7 / 76.4 -> 79.4 tok/s (the kernel switches alone 78.3). quant_quality against the same engine without them: top-1 100.00%, KL 0.00000.

**2. The sm_120 decode switches on sm_89** (engine 0.1.63, `STRATA_L2_PF=1 STRATA_Q80_ROWS1=1 STRATA_EXP_MODE=8 STRATA_HC_UPMIX=1`, 2 paired rounds)

| | without | with |
|---|---|---|
| UD-Q4_K_XL greedy decode | 92.2 / 91.7 | 95.2 / 96.0 (+4.0%) |
| UD-Q4_K_XL 34K prompt | 4,777 tok/s | 4,867 (+1.9%) |
| UD-Q3_K_XL greedy decode | 141.7 / 144.8 | 145.5 / 143.3 (+0.8%) |
| UD-Q3_K_XL 34K prompt | 5,878 tok/s | 6,084 (+3.5%) |

Not bit-exact here: against the run without them, top-1 98.0%, KL 0.017 (6K positions of agent transcripts). Distance to a UD-Q4_K_XL reference unchanged (KL 0.2593 with vs 0.2603 without; 2 of 4 segments identical).

**3. The CPU prompt share for agent follow-up turns** (engine 0.1.63; a 20K-token conversation, then tool-result-sized turns; 2 rounds each)

The 0.2.17 auto rule only raises the share on 12.0 cards. With `STRATA_PREFILL_TIMING=1`, "wait copy" is 48-63% of a 300-3,000-token follow-up turn's prompt time here (e.g. 1,015 tokens: 690 of 1,146 ms).

| time to first token after a tool result of | 300 tok | 1,000 tok | 3,000 tok | fresh 4.8K prompt |
|---|---|---|---|---|
| UD-Q4_K_XL default share | 0.99 s | 1.66 s | 3.31 s | 3.16 s |
| UD-Q4_K_XL `STRATA_PREFILL_CPU_SHARE_AUTO=1` | 0.97 s | 1.61 s | 3.11 s | 3.12 s |
| UD-Q4_K_XL `_NT=32 _MAX=6144` | 0.96 s | 1.60 s | **2.44 s** | **2.59 s** |
| IQ3_S default share (0.1.63) | 0.71 s | 1.10 s | 2.00 s | 2.18 s |
| IQ3_S `_NT=32 _MAX=6144` | 0.66 s | 1.07 s | 1.86 s | 1.93 s |

`_NT=64 _MAX=12288` was not better than 32 / 6144 on UD-Q3_K_XL (0.74 / 1.19 / 1.95 s vs 0.73 / 1.20 / 1.88 s). Prompts of 16K and 34K and decode are unchanged by the share. `AUTO=1` printed no link-based change here (DMA PCIe mode).

**4. Smaller items on this box**
- `--pool-workers 23` (7 P-cores + 16 E-cores, host thread on CPU 0) vs setup's 15: UD-Q4_K_XL 76.6 -> 80.5 / 79.8; 11 (auto) 71.9; P-cores only 71.5; SMT siblings 70.0.
- `--pcie-frac 0` vs the auto share (`--pcie-mode dma`): UD-Q4_K_XL 83.1 / 82.3 -> 90.1 / 89.4 (+8%).
- `STRATA_ROUTE_TAIL_SKIP=7` (0.2.17 default off): real-sampling decode IQ3_S 109 -> 125, UD-Q3_K_XL 105 -> 118.5 (greedy 122.5 / 123.4 -> 145.3 / 144.0). Drift vs a UD-Q4_K_XL reference on 36K positions of real Claude Code transcripts: IQ3_S 0.290 -> 0.301, UD-Q3_K_XL 0.243 -> 0.271.
- `STRATA_ARENA_THP=1`: UD-Q4_K_XL +3.9% greedy / +3.8% real; IQ3_S level. `STRATA_POOL_DEP=0`: level.
- 0.1.63 vs 0.1.62, same settings: about -2% decode on every model (IQ3_S real 124.8 vs 127.4; UD-Q4_K_XL 76.6 vs 78.2); not explained by `POOL_DEP` or the tail skip.
- Decode vs context, 0.1.63 (real sampling, a growing conversation of real code/docs, 2 rounds): IQ3_S 156 / 142 / 151 / 154 / 147 tok/s, UD-Q3_K_XL 151 / 153 / 140 / 161 / 142, UD-IQ3_XXS 170 / 157 / 152 / 168 / 152 at 42K / 81K / 119K / 159K / 201K tokens.

Suggestion: an sm_89 row in `nv_defaults` with the bit-exact sm_86 set (1.), and letting the CPU share's raised limit (3.) apply to sm_89 single cards, or documenting `_NT=32 _MAX=6144` for them.

### Reproduction status

Measured locally

No site

Links install, modelos, releases.