#6616 Engine 0.1.65 (NVFP4 GPTQ pack): EXP_MODE on = 0 drafts accepted for every drafter; --prefill 32768 = NaN at 24k

open · @sdbm007 · 2 comments · View on GitHub

needs-format

Description

**Area:** Engine

**Desktop app, Terminal or source?:** Terminal (`strata launch`, official launcher) and a hand-written run config; both reproduce.

**Reproduction status:** Reproduced (every start, 5+ times).

**Strata version:** 0.2.19, engine 0.1.65 `windows-x86_64-cuda-cuda13` (`f609f862fa9e+cuda13.0`), Control 0.1.65.

**Graphics card, memory and system:** RTX 5090 D 32 GB (sm_120), driver 621.14, Windows 11, 127.7 GB RAM, 24 threads.

**Model:** `Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` (the "NVFP4 (Strata's version)" entry in `docs/NVFP4.md`), native pack + native dense + PLE FP8 + `expert-profile.bin`. Files match the repository's own `SHA256SUMS` (pack/index.txt, pack/native_experts.txt, pack/dense.bin, tokenizer; the MTP files match too).

---

Two reproducible problems. Both also occur through the **official launcher** with its own defaults, so they are not artifacts of a hand-written config.

## 1. With the default sm_120 switches, every drafter is accepted 0 times

Start line: `sm_120 (Blackwell): 12 exact speed switches on by default (... EXP_MODE ...)`.

| drafter | pack | drafts accepted | decode |
| --- | --- | ---: | ---: |
| DFlash2 (legacy Q8 pack, 760 MB) | legacy | **0 of 216 / 0 of 293 / 0 of 365** | 53-87 |
| DFlash2 (tuned `strata1.q4_0-selq4`, sha `4df2e5ab…`) | tuned | **0 of 133 / 0 of 454** | 45-72 |
| MTP (official `mtp/` + `rt-tuned/`) | mtp | **0 of 127 / 0 of 800** | 40-63 |

With `STRATA_EXP_MODE=0` (only that switch off) the same MTP drafter accepts again: 305/681, 298/685, 329/756 (45-48%), decode 90-97.

A one-switch-at-a-time sweep found only `EXP_MODE`: baseline 0/61, `HC_UPMIX` 0/81, `L2_PF` 0/69, `Q80_ROWS1` 0/81, `SIDE_MIXER` 0/32, `DEQUANT_SIDE` 0/173, `QFUSE` 0/76, **`EXP_MODE` 37/60**. The engine loads the drafter correctly (`selector codebooks: Q4_0 rows`, `windows up to 6 tokens`) and still never accepts a draft.

Same machine, same pack: **0.1.59 accepts the same DFlash2 drafts at ~40% and decodes 145-172 tok/s** (0.1.62: ~40%, 139-159).

**Official launcher check** (same pack, engine 0.1.65, defaults chosen by setup: 128K, `--prefill auto`, MTP):

```
strata launch --model nvfp4 --family nvfp4 --context 128k --vision gpu --tune skip --access local --port 8081 --yes --no-tui
```

served normally, three long generations: **drafts accepted 0 of 474, 0 of 633, 0 of 82**; decode **42.4 / 43.7 / 69.4 tok/s**.

## 2. `--prefill 32768` gives non-finite output on a 24k prompt

```
the model's output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state - send the request again
```

Reproduced on consecutive requests, with `STRATA_ROUTE_TAIL_SKIP` 0 and 7, with and without `--kv`, with and without YaRN. With `--prefill 8192` the same prompt answers every time (2.7k tok/s prompt read); with 32768 the prefill reaches 5.7k tok/s before generation turns non-finite.

This is close to #6584 (`EXP_MODE`/`PREFILL_NVFP4` on another NVFP4 pack), so it may be the same area. Happy to run any build or collect traces.

Related on this site

Quick links to related pages.