#2239 sm_75 (2x RTX 2080 Ti): GDN 'hc0 norm' on GPU0 takes 135-213 ms/window (GPU1: 0.05 ms) — decode stuck at ~10 tok/s

Description

## Summary

On a 2× RTX 2080 Ti (sm_75) layer split, decode of Qwen3.8-Flash-Next IQ3_XXS tops out at ~8–10 tok/s (expected ~40+ from the 12 GB-card reference numbers scaled to a 99% VRAM expert-cache hit rate). `STRATA_DECODE_TIMING=1` + `STRATA_VERIFY_PROFILE=1` show one line item eating 92% of the verify stage: **`hc0 norm` in GPU0's GDN layers takes 135–213 ms/window, while the same stage in GPU1's GDN layers takes 0.05–0.22 ms**. Everything else in both stages sums to ~35 ms/window.

## Environment

- GPU: 2× NVIDIA RTX 2080 Ti 22 GB (sm_75), driver 615.71.09, PCIe 3.0 x16 (probe: 9.1–9.6 GB/s host→device), NVLink bridge present
- CPU: Xeon E5-2650L v2 (Ivy Bridge: AVX, **no AVX2** → experimental `STRATA_ISA_FLOOR=avx` build)
- RAM: 157 GB DDR3-1600
- OS: Arch Linux, kernel 7.2.6-arch2-1
- Engine: strata 0.1.43, built from source at 7695859 (2026-10-11) with CUDA 13.4, gcc-13, `-DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_ISA_FLOOR=avx`
- Model: Qwen3.8-Flash-Next IQ3_XXS (native pack), MTP on, max-context 131072, KV int8 with 32768 resident cells, `--layer-split auto` → K=23 (GPU0 layers [0,23), GPU1 [23,48) + head + draft)

## Measurements

Decode ~7.0–10.2 tok/s across story/code/Chinese prompts (drafts accepted 55–70%). Expert cache hit rate 94–99.6% (20491/24576 profiled pairs in VRAM), misses served over PCIe — so this is not expert-miss or CPU-bound. Prefill is fine: 5,065-token prompt at 640 tok/s.

Default build (`hc` staged path), 300-token generation:

```
strata decode timing: 165 windows, avg T 2.41, 1.82 tokens/window, 258.78 ms/window = verify 255.29 (GPU-reach wait 0.00 + per-layer host 0.00 [plan 0.27 actq 0.12 jobs 0.04 CPU 3.51] + stage 221.91) + commit/emit 0.69 + draft 2.20; per layer-window: CPU experts 0.20 (0.26 entries), VRAM hits 12.07, PCIe 0.01
strata decode GPU stages (ms/window): GDN layers: q8+qkv/q-idx gemv 1.03 conv 0.08 ab 0.13 z 0.70 rec 0.25 out-proj 0.68 router+plan+bell 0.33 shared+quant 0.61 waitA 0.02 VRAM hits 2.29 copy+combine 0.07 (gap) 0.02   hc0 norm 213.39   hc0 down 0.41   hc0 up 0.45   hc1 norm 0.10   hc1 down 0.40   hc1 up 0.45 | QSA layers: ...   hc0 norm 0.03 ... | total 223.92 ms/window over 177 windows
strata decode GPU stages, stage 1 (ms/window): GDN layers: ...   hc0 norm 0.13   hc0 down 0.49   hc0 up 0.53 ... | total 22.07 ms/window over 177 windows
```

Same build restarted with `STRATA_HC_SPLIT=0` (the plain read):

```
strata decode timing: 163 windows, avg T 2.40, 1.84 tokens/window, 180.41 ms/window = verify 177.09 (GPU-reach wait 0.00 + per-layer host 0.00 [plan 0.22 actq 0.11 jobs 0.03 CPU 3.30] + stage 144.32) + commit/emit 0.62 + draft 2.13; ...
strata decode GPU stages (ms/window): GDN layers: ...   hc0 norm 135.57   hc0 down 0.56   hc0 up 0.45 ... | total 146.76 ms/window over 175 windows
strata decode GPU stages, stage 1 (ms/window): GDN layers: ...   hc0 norm 0.22 ... | total 21.15 ms/window over 175 windows
```

`STRATA_HC_SPLIT=0` helps (258.8 → 180.4 ms/window, 7.0 → 10.2 tok/s) but `hc0 norm` on stage 0 remains the wall at 135.6 ms.

## Observations

- The asymmetry is extreme: identical op, ~0.1 ms on stage 1's GDN layers (GPU1), 135–213 ms on stage 0's (GPU0). Same GPU model, same driver.
- GPU0's other GDN stages look normal (q8+qkv gemv 1.03, z 0.70, out-proj 0.68, VRAM hits 2.29 ms), so this doesn't look like a general "GPU0 is slow" problem.
- GPU0 utilization reads 90–96% during decode at only ~105–125 W (of the 300 W cap) at full SM clock — lots of small work or spinning, not a compute-saturated kernel.
- `--spec 6` instead of 4 does not help (window time, not window count, is the bottleneck).
- Possibly the `hc0 norm` timing slot on stage 0 includes a cross-device wait (stage boundary / doorbell?) rather than pure kernel time — but then 135 ms of per-window wait on a 2-stage split would still be the anomaly.

## Questions

1. Is this a known sm_75 (or 2-stage layer split) issue? Any knob beyond `STRATA_HC_SPLIT` I should try?
2. If it helps, I can rerun with extra instrumentation (e.g. `STRATA_DECODE_TIMING`, layer dumps, a forced `--layer-split 26`, or single-GPU) — happy to provide numbers or test a patch.

Thanks for the engine — first time this box has ever run a 125 B model at all, and the answers look right. Even at 10 tok/s it's usable; fixing this would put it in daily-driver territory.