#6616 Engine 0.1.65 (NVFP4 GPTQ pack): EXP_MODE on = 0 drafts accepted for every drafter; --prefill 32768 = NaN at 24k
open · @sdbm007 · 1 comentários · No GitHub
Descrição
Two reproducible problems on **0.1.65 with the NVFP4 GPTQ pack** (`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata`, the "NVFP4 (Strata's version)" entry in `docs/NVFP4.md`). This continues #6074 (prompt-read regression on the same pack), but these are 0.1.65-specific and separate from that one. **Environment:** Windows 11, RTX 5090 D 32 GB, driver 621.14, engine `0.1.65 windows-x86_64-cuda-cuda13` (`f609f862fa9e+cuda13.0`), Control 0.1.65. Pack: native pack + native dense + PLE FP8 + expert-profile, `--expert-cache auto`. Args: `--max-context 524288 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144 --kv k8v4 --prefill 8192 --spec 6 --vision`. Every measurement is a cold start, 2 warm-up requests, then the measured request. ## 1. With the default sm_120 switches, **every drafter is accepted 0 times** The start line says 12 exact speed switches are on by default on sm_120 (`HC_UPMIX L2_PF Q80_ROWS1 SIDE_MIXER DEQUANT_SIDE QFUSE LFUSE PLE_COMPACT PLE_COPY_STREAM PREFILL_CPU_SHARE_NT PREFILL_NVFP4 VERIFY_DEVICE_HITS`). | drafter | pack | drafts accepted | decode | | --- | --- | ---: | ---: | | DFlash2 (legacy Q8 pack, 760 MB) | legacy | **0 of 216 / 0 of 293 / 0 of 365** | 53-87 | | DFlash2 (tuned `strata1.q4_0-selq4`, 339.7 MB, sha `4df2e5ab…`) | tuned | **0 of 133 / 0 of 454** | 45-72 | | MTP (official `mtp/` + `rt-tuned/`, after fixing the missing `dense.txt`) | mtp | **0 of 127 / 0 of 800** | 40-63 | With `STRATA_EXP_MODE=0` (only that switch off, the other 11 as they were) the same MTP drafter accepts again: | setting | drafts accepted | decode | | --- | ---: | ---: | | `EXP_MODE` on (default) | 0 of 127 | 40-63 | | `EXP_MODE=0` | 305 of 681, 298 of 685, 329 of 756 (45-48%) | 90-97 | A one-switch-at-a-time sweep on the same config found only `EXP_MODE` (baseline 0/61, `HC_UPMIX` 0/81, `L2_PF` 0/69, `Q80_ROWS1` 0/81, `SIDE_MIXER` 0/32, `DEQUANT_SIDE` 0/173, `QFUSE` 0/76, **`EXP_MODE` 37/60**). The engine loads the drafter correctly (`strata dflash2: ... drafter 688 MiB (Q4_0 weights)`, `selector codebooks: Q4_0 rows`, `windows up to 6 tokens`), so the drafts are produced and never verified. On the same machine and pack, **0.1.59 accepts the same DFlash2 drafts at ~40% and decodes at 145-172 tok/s**, so this is not the pack/tokenizer. I could not get 0.1.65 above ~97 tok/s here: `EXP_MODE=0` is required for any acceptance, and it costs decode. ## 2. `--prefill 32768` gives non-finite output on a 24k prompt With `--prefill 32768`, a single 24,647-token prompt (single chunk) returns HTTP 503: ``` the model's output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state - send the request again ``` Reproduced on consecutive requests, with `STRATA_ROUTE_TAIL_SKIP` 0 and 7, with and without `--kv`, with and without YaRN. With `--prefill 8192` the same prompt answers every time (2.7k tok/s prompt read); with 32768 the prefill itself reaches 5.7k tok/s before the generation turns non-finite. Questions: is `EXP_MODE` expected to interact with draft verification on this pack (i.e. is 0 acceptance "expected" and the switch should be documented as off for NVFP4)? And is a 32768 prefill chunk supported for the NVFP4 pack, or should setup clamp it? Happy to run any build or collect more logs.
No site
Links install, modelos, releases.