#3328 NVFP4 pack: deterministic token soup on an AVX2-only CPU (Core Ultra 9 285K) + RTX PRO 6000 (sm_120) — every expert path, every documented fallback

本文

## Environment

- Windows 11 Pro, engine 0.1.45 release (E:\Strata, engine built from the release; `strata.exe` in `engine\`, BUILD.json present)
- CPU: Intel Core Ultra 9 285K (Arrow Lake — **no AVX-512**)
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation, 96 GB (sm_120)
- RAM: 128 GB DDR5-6400
- Model: `jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`, converted with the documented
  pipeline (`tools/nvfp4_convert.py` with the pinned llama.cpp, then `tools/iq_pack.py`), run per docs/NVFP4.md
  with `--ple-gguf` pointing at Unsloth's UD-IQ4_XS shard 2 and the stock MTP runtime.

## Symptom

Greedy decoding (temperature 0) produces deterministic token soup starting from the **first generated token**,
on every configuration tried:

| config | decode | soup opening |
| --- | --- | --- |
| `--expert-cache auto` (24,000 GPU-resident) | 114 tok/s | `Guardty (全球的 gr deviltyty grad tslint...` |
| same + `STRATA_PREFILL_NVFP4=fp16` | soup | identical stream to `w4a8` for the full 100 tokens |
| same + `STRATA_PREFILL_NVFP4=w4a4` | soup | identical stream to `w4a8` and `fp16` for the full 100 tokens |
| `--expert-cache 1` (all-CPU pool, AVX2 rows) | soup | `Guardty ( blotyty gr格selcapcb off...` |
| all-CPU pool + `STRATA_NO_NVFP4_256=1` | soup | identical to the AVX2-row run (expected: the doc specifies AVX2 rows are bit-equal to ggml-cpu, so this only rules out a row-kernel numeric bug, not wrong input bytes) |
| no MTP (`--mtp` dropped) | soup | identical to the GPU-cache run |

Env delivery was verified with a sentinel: with `STRATA_ADAPT_OVERLAP=1` in the same config `env` block, the
startup log flips from `adaptive overlap: off (...)` to the enabled wording, so config-borne `STRATA_*` variables
do reach this engine build (both switches are also present as strings in `strata.exe` 0.1.45).

The `w4a8` / `w4a4` / `fp16` result is the sharpest observation: three numerically very different product paths
(int8 MMQ, Blackwell FP4 MMA, dequantize + cuBLAS FP16 — the doc's own numbers put `w4a4` ~5x further from
reference than `w4a8`) produce **byte-identical** 100-token argmax streams. Whatever is broken sits upstream of
where these modes diverge — i.e. in a stage all three share (host-side blob preparation, expert selection, or the
non-expert skeleton as fed by this model) — not in any one kernel's arithmetic.

Yardstick (`tools/quant_quality.py`, 4,995 teacher-forced positions vs the same engine serving
Unsloth UD-IQ4_XS of the base model): top-1 agreement **0.00 %**, mean KL 3.27, PPL **434,672**
(reference model: PPL 4.17). The same engine run of the UD-IQ4_XS model on this machine is completely normal
(207 tok/s decode, sane output, yardstick reference numbers above) — so the shared skeleton, KV, sampling and
serving paths are fine; the failure is specific to the NVFP4 expert path.

## What I verified about the data (all green)

1. `tools/nvfp4_verify_gguf.py --model <ckpt> --gguf orca-nvfp4.gguf --ple-gguf <UD-IQ4_XS shard2> --samples 300`
   → **RESULT: OK** (experts bitwise + scale order, dense policy `F32 x580, BF16 x484, Q8_0 x302, NVFP4 x144, F16 x1`,
   PLE cosine 0.9967 worst 0.9958 over 64 rows vs the borrowed table).
2. Independent checks beyond the tool:
   - expert **order**: `ffn_down_exps.input_scale` (varies per expert) matches the checkpoint's per-expert
     `input_scale` **identity 512/512** on blk.0 — no string-sort vs router-row mismatch;
   - **full dense sweep, all 48 layers**, every non-expert tensor vs the checkpoint with the converter's
     transforms applied (V-reorder for qkv/z/conv, ssm_out column perm, index_qk split, +1 zero-centred gammas):
     Q8_0 tensors rel_rms ≤ 0.006, BF16/F32 exact; the flagged ones from my first pass were my own missing `+1`
     (`got == ref + 1` to the last bit, i.e. the converter is right);
   - `experts.bin` byte-identical to the GGUF expert slices incl. the 16-byte `{s_gate, s_up, s_down, 0}` tails;
   - the checkpoint's PLE hash constants are exactly the engine's `ple_artifact_consts()` (ngram.cpp);
   - PLE row geometry matches (320,001,536 rows; heads cover 320,001,446, padded to 128).

## Why I suspect the host-side blob preparation on AVX2-only machines

- This is the first machine in the doc's test matrix I can find that ran the **full model** with
  **no AVX-512**. The log says: `this CPU has no AVX-512: the expert kernels run on AVX-2`.
  The binary also carries `STRATA_FORCE_AVX2`, which suggests the AVX2 path is normally exercised on AVX-512
  hosts via that switch rather than being the daily driver on a no-AVX-512 machine.
- The soup differs between the CPU-pool run and the GPU-cache run (two different wrong results), which fits
  two different consumers of a common, wrongly-prepared input — but both the GPU cache fill and the CPU pool
  read the same `experts.bin`.
- The parity tests (`nvfp4_cpu_parity`, `mmq_nvfp4_parity`, `nvfp4_expert_gpu_parity`) compare kernels fed the
  **same** input bytes; a selection/packing bug upstream of the row kernels (wrong slice, wrong tail-scale
  association, wrong nibble/sub-scale order) would pass all of them while poisoning every consumer. The
  `STRATA_NO_NVFP4_256` fallback producing bit-identical output to the AVX2 rows is consistent with the doc's
  "bit-equal to ggml-cpu" design and therefore only tells us the row arithmetic is faithful to whatever bytes
  it is given.
- `STRATA_PREFILL_NVFP4=fp16` (dequantize + cuBLAS) producing the *same* opening as `w4a8` points at the blob
  contents rather than the products: two very different kernel paths, one wrong result, deterministic.

## What would help

- Pointers to any dump switch that would let me capture the engine's decoded NVFP4 blocks (weights and tail
  scales) for one expert, so I can diff them against `tools/nvfp4_verify.py`'s dequant of the same blob;
- whether the AVX2 (no-AVX-512) path has been validated **end to end** (full generation) anywhere, as opposed
  to row-level parity;
- any known interaction with Arrow Lake specifically (the doc's AVX2 list includes Core Ultra 7 265, but those
  results are `nvfp4_cpu_parity` row counts, not full-model runs).

Happy to run any build/test you push to the repo — I just can't compile locally (no CUDA toolchain on this box).