Pull requests / #3384

#3384 [Engine] run Mirai's Qwen3.8-Flash-Next-S (QNS trellis) packs

open · draft · @alesha-pro · 0 comentários · No GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Descrição

## Problem and result

Mirai published Qwen3.8-Flash-Next-S, a trellis build of Flash-Next (routed experts 1-5 bits each, 2.4 on average; backbone and head 5 bits), packed for Strata: `trymirai/Qwen3.8-Flash-Next-S-strata-experimental`. Until now it runs only on v0.1.43 with Mirai's `strata-qns.patch`. This PR ports that patch to 0.2.0: the S and S-small packs load and run on one card and on a layer split, with MTP and the refusal projection. Every other model takes the same path as before, checked on IQ3_S below.

## Related discussion

Scope: I am writing to @Niko1221 directly to agree it. The QNS format, the CUDA and AVX2 kernels and the packer support are Mirai's work, from their `strata-qns.patch`.

## Approach

Mirai's code is taken unchanged from their HF repo (2026-10-15, which already has the layer-split rotation fix). What the merge with 0.1.44-0.2.0 needed:

- **Pack version.** 0.2.0 uses `native_experts.txt` v5 for Hadamard-folded experts, and the patch used v5 for QNS layers. QNS packs are v6 here. v5 keeps its meaning (hadamard.txt is required for v5 only). A v5 QNS pack from the old patch is refused with "repack with tools/iq_pack.py"; repacking takes ~20 s and gives the same files except the header line. The packer skips the Hadamard spec for QNS.
- **MTP head on another card** (0.1.44 moved the drafter to the first stage): the trellis head's subset rows and their row scales are gathered on the head's card and copied over, like the non-QNS path.
- **New side-stream paths** that read dense weights directly now take the trellis GEMV for `kQnsDenseType`: the GDN gate projection (`side_proj`) and the QSA query chain on a branch stream.
- Expert cache sizing uses `slot_stride(expert_bytes(l, e), slot_guard)`; `expert_bytes` is the layer's blob for non-QNS packs.
- The CPU pool keeps 0.2.0's per-layer widths and rounds QNS row ranges to whole 8-row blocks.
- `docs/QNS.md`: how to pack and run.

All new code is behind `expert_layout().qns()` or `kQnsDenseType`.

## Validation

Ubuntu 22.04, CUDA 12.8, 4x RTX 3090 at 300 W, EPYC 7642 (AVX2), 125 GB RAM. Branch head 8e75f6f (engine sources identical to the tested build).

**Passed**
- CUDA build: `cmake -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DSTRATA_BUILD_TESTS=OFF`, `--target strata`. No new warnings in the touched files.
- `tools/iq_pack.py` on S and S-small: v6 packs; `qns_offsets.bin`, `qns_rotation.bin`, `dense.bin`, `index.txt` and every layer row byte-identical to the v0.1.43 patch's packs.
- Six greedy probes through `/v1/chat/completions`, thinking off, 32K context, MTP on, the refusal vector on: a sum, a Python function, a Russian answer, a tool call, a 6,210-token document with a hidden code, a lock-picking question.
  - S-small, 1 card and 2 cards (layer split): all answers correct; the code is found; the tool call is right; no refusal. Against v0.1.43 + Mirai's patch on the same cards: 5 of 6 answers identical, the sixth parts on a near-tie after 10-55 words (v0.1.43 itself gives a different text on 1 and 2 cards there, from about 120 words on).
  - S (full), 1 card and 2 cards: all six correct.
  - On 2 cards MTP drafts are accepted (117 of 170 on a 200-token answer), no errors in the engine log.
- **Default path:** IQ3_S (ISTA GSQ-RCO) on 2 cards, the same flags, stock 0.2.0-beta (3d1fb48) against this branch: all six greedy answers byte-identical, same token counts; decode per request within 3.1%.
- Decode, server tok/s per request (one run each, indicative): S-small 47-106 (1 card) / 83-166 (2 cards); S 35-81 / 60-125. v0.1.43 + patch on the same S-small requests: 45-92 / 75-143.

**Failed:** none observed.

**Not run**
- HIP build: no ROCm toolchain here. The new `.cu` files are compiled for HIP by the existing glob, so this needs a HIP build (or a CUDA-only guard) before review. Help welcome.
- SYCL and Metal: not touched. Windows / MSVC: not built.
- Long contexts (above 32K) and vision on QNS: not run in this PR (vision ran on v0.1.43 with the patch).

## Limits

- NVIDIA CUDA and the AVX2 CPU pool only, as in Mirai's patch.
- Setup and the Terminal do not offer these models; they start from a hand-written config (`docs/QNS.md`).
- Kept as a draft until the HIP build is checked.

No site

Links install, modelos, releases.