#6616 Engine 0.1.65 (NVFP4 GPTQ pack): EXP_MODE on = 0 drafts accepted for every drafter; --prefill 32768 = NaN at 24k
open · @sdbm007 · 1 Kommentare · Auf GitHub
Beschreibung
Two reproducible problems on **0.1.65 with the NVFP4 GPTQ pack** (`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata`, the "NVFP4 (Strata's version)" entry in `docs/NVFP4.md`). This continues #6074 (prompt-read regression on the same pack), but these are 0.1.65-specific and separate from that one. **Environment:** Windows 11, RTX 5090 D 32 GB, driver 621.14, engine `0.1.65 windows-x86_64-cuda-cuda13` (`f609f862fa9e+cuda13.0`), Control 0.1.65. Pack: native pack + native dense + PLE FP8 + expert-profile, `--expert-cache auto`. Args: `--max-context 524288 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144 --kv k8v4 --prefill 8192 --spec 6 --vision`. Every measurement is a cold start, 2 warm-up requests, then the measured request. ## 1. With the default sm_120 switches, **every drafter is accepted 0 times** The start line says 12 exact speed switches are on by default on sm_120 (`HC_UPMIX L2_PF Q80_ROWS1 SIDE_MIXER DEQUANT_SIDE QFUSE LFUSE PLE_COMPACT PLE_COPY_STREAM PREFILL_CPU_SHARE_NT PREFILL_NVFP4 VERIFY_DEVICE_HITS`). | drafter | pack | drafts accepted | decode | | --- | --- | ---: | ---: | | DFlash2 (legacy Q8 pack, 760 MB) | legacy | **0 of 216 / 0 of 293 / 0 of 365** | 53-87 | | DFlash2 (tuned `strata1.q4_0-selq4`, 339.7 MB, sha `4df2e5ab…`) | tuned | **0 of 133 / 0 of 454** | 45-72 | | MTP (official `mtp/` + `rt-tuned/`, after fixing the missing `dense.txt`) | mtp | **0 of 127 / 0 of 800** | 40-63 | With `STRATA_EXP_MODE=0` (only that switch off, the other 11 as they were) the same MTP drafter accepts again: | setting | drafts accepted | decode | | --- | ---: | ---: | | `EXP_MODE` on (default) | 0 of 127 | 40-63 | | `EXP_MODE=0` | 305 of 681, 298 of 685, 329 of 756 (45-48%) | 90-97 | A one-switch-at-a-time sweep on the same config found only `EXP_MODE` (baseline 0/61, `HC_UPMIX` 0/81, `L2_PF` 0/69, `Q80_ROWS1` 0/81, `SIDE_MIXER` 0/32, `DEQUANT_SIDE` 0/173, `QFUSE` 0/76, **`EXP_MODE` 37/60**). The engine loads the drafter correctly (`strata dflash2: ... drafter 688 MiB (Q4_0 weights)`, `selector codebooks: Q4_0 rows`, `windows up to 6 tokens`), so the drafts are produced and never verified. On the same machine and pack, **0.1.59 accepts the same DFlash2 drafts at ~40% and decodes at 145-172 tok/s**, so this is not the pack/tokenizer. I could not get 0.1.65 above ~97 tok/s here: `EXP_MODE=0` is required for any acceptance, and it costs decode. ## 2. `--prefill 32768` gives non-finite output on a 24k prompt With `--prefill 32768`, a single 24,647-token prompt (single chunk) returns HTTP 503: ``` the model's output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state - send the request again ``` Reproduced on consecutive requests, with `STRATA_ROUTE_TAIL_SKIP` 0 and 7, with and without `--kv`, with and without YaRN. With `--prefill 8192` the same prompt answers every time (2.7k tok/s prompt read); with 32768 the prefill itself reaches 5.7k tok/s before the generation turns non-finite. Questions: is `EXP_MODE` expected to interact with draft verification on this pack (i.e. is 0 acceptance "expected" and the switch should be documented as off for NVFP4)? And is a 32768 prefill chunk supported for the NVFP4 pack, or should setup clamp it? Happy to run any build or collect more logs.
Mehr auf der Site
Links zu Install, Modellen, Releases.