#6510 [Engine] --experts-wex4: read the routed experts from a checked WEX4 container (opt-in, file tier)

closed · draft · @KairaCode · 0 comentarios · En GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

## Problem and result

A native pack's routed experts live in one big file (`experts.bin`, 25 GB for Qwen3.8-Flash-Next Coder IQ1_M) or are read from the GGUF in place. Nothing records a hash of those expert bytes: setup checks each shard's SHA-256 once after the download, and after that a flipped byte in a copied or ageing file goes unnoticed until the answers get strange.

This PR adds an opt-in way to keep the experts in a **WEX4 container**. Think of a ZIP file with compression turned off: the same bytes, but every expert is its own entry with its own SHA-256, and the SHA-256 of the 4 KB header page (the "root") names the whole file. Nothing is converted or re-quantized, so the model and its output do not change.

- `iq_pack.py --experts-wex4` writes `experts.wex4` beside the pack.
- `terminal/tools/wex4_check.py FILE` hashes every expert once and names the layer and expert of any damaged one (20 s for 25 GB here).
- `strata generate ... --mmap-experts --experts-wex4 FILE [--experts-wex4-root HEX]` reads the experts from the container. At every start the engine checks the header, the index and that the container matches the pack's expert layout (28-39 ms); it does not hash the 25 GB at start.

Without `--experts-wex4` nothing changes: same files, same offsets, same arithmetic (checked below).

## Related discussion

None yet. This is a new opt-in feature, so I am opening it as a draft to ask whether you want it in scope; happy to move it to an issue first, cut it down (for example the engine flag alone), or drop it.

## Approach

- **One record per expert, holding exactly the `experts.bin` blob** `[gate | up | down]`. So only a blob's file offset changes: expert `e` of layer `l` is at the layer's first record + `e` x the record size (the blob rounded up to 4096). `FileExpertSource` keeps that size per layer (`layer_stride_`); without the flag it equals the blob size, so the existing paths compute the same offsets from the same numbers. The offset sites are `mapped_blob`, the two `read_direct` paths (Windows and Linux) and the Linux layer release (`madvise` / `posix_fadvise`); everything else goes through them.
- **A small reader of its own** (`engine/src/core/wex4_experts.cpp`, RAW records only, portable SHA-256 for the header page and the index). Every field read from the file is bounds-checked before use: header and index SHA-256, version and feature bits, file size, index size and position, name table offsets, record and tensor indexes, each record's codec, tensor names, dtypes, sizes, alignment, and that a layer's records are consecutive and inside the file. A container of a bigger model (an extra expert) is refused, not read in part. Each refusal starts with `--experts-wex4:` and names the file, layer, expert and rule.
- **Refused by name:** without the file tier (`--mmap-experts` / `--resident-experts`), in the Metal engine, a canonical pack, NVFP4 layers with a scale tail, ggml types above 127 (Gyro's trellis types), records stored with a compressed codec (WNR3, WBP1).
- **Tools:** the WEX4 reference reader/writer `wex4.py` (Python standard library only, Apache-2.0) is vendored unchanged in `engine/third_party/wex` with its `LICENSE` and `NOTICE`, pinned in `UPSTREAM.md` and checked by `tools/check_vendored.py`. The engine does not use it. The Terminal bundle carries `third_party/wex` because `iq_pack.py` and `wex4_check.py` import it.
- **Docs:** `docs/WEX4.md` (what is checked when, the costs, the record convention, the limits), linked from `docs/README.md`.
- SYCL keeps its own copies of `expert_source.cpp` and `generate.cpp` and is not changed. Metal gets the new source file in its list and refuses the flag.

## Validation

Revision: branch `wex4-experts` on top of `3142b66` (main). Engine built with MSVC 2022 BuildTools / Ninja / CUDA 13.5, `CMAKE_CUDA_ARCHITECTURES=120`, `STRATA_BUILD_TESTS=ON`. Rig: RTX 5090 32 GB, Ryzen 9 7950X3D, 32 GB RAM (about 20 GB free during the runs), files on one 1 TB SK hynix NVMe, Windows 11. Model: Qwen3.8-Flash-Next Coder IQ1_M (the two ISTA shards), packed with `iq_pack.py`; `experts.bin` written with `--experts-bin`, `experts.wex4` with `--experts-wex4`. "Base" is the engine built from `3142b66` without this change.

**Passed:**
- `wex4_experts_test` (CPU, registered in `cpu.cmake`): SHA-256 against two FIPS 180-4 vectors; a valid container's offsets and every expert's bytes; the root pin; damaged header page, damaged index, truncated file; a damaged expert payload is NOT caught at open (documented); a WNR3 record, tensors out of order, wrong dtype, a gap between records, a missing expert, an extra expert, a canonical pack, an NVFP4 tail, a type above 127 refused. PASS with MSVC; also compiled with MinGW g++ 16.2 `-std=c++20 -Wall -Wextra -Wpedantic` and run: PASS, no warning in the new files.
- `file_expert_source_test` (CUDA build, on the 5090): one pack read from `experts.bin` and from WEX4 gives the same bytes through the mapping, the unbuffered reads (`read_direct` merging a layer's records across their padding) and `copy_blob`; the session inputs name the container; a wrong root refuses to open. PASS.
- `wex4_check.py experts.wex4`: `ok, 12288 records intact`, 25.17 GB in 20.0 s and 20.2 s. All 12,288 WEX4 records compared byte for byte with the `experts.bin` written from the same GGUF: equal.
- Output unchanged: `quant_quality.py run` (5 segments x 255 positions, experts on the CPU from the file tier): the per-position log-probabilities of the text token and the top 64 are byte-identical across six runs: experts.bin twice, WEX4 twice, the GGUF in place once, and base on experts.bin once (so the default path is unchanged too).
- Generate, 999-token prompt + 128 tokens greedy, `--mmap-experts --spec 2 --suffix-draft 0 --prefill 512 --expert-cache 1000`, `STRATA_ROUTE_TAIL_SKIP=0`: all 27 runs (branch on experts.bin, branch on WEX4, base, GGUF in place) gave the same 128 tokens. Decode tok/s, two series, each arm three times (series 2 runs each arm twice in a row and counts the second run, so each counted run starts with its own file in the page cache):

  | Series | experts.bin decode | WEX4 decode | experts.bin prompt | WEX4 prompt |
  | --- | --- | --- | --- | --- |
  | 1 | 30.28, 28.53, 37.88 | 31.55, 31.34, 37.22 | 70.47, 76.75, 76.28 | 65.38, 74.37, 70.88 |
  | 2 | 21.66, 25.83, 31.86 | 25.99, 24.09, 31.66 | 73.47, 71.36, 61.27 | 75.44, 74.88, 63.07 |

  In these runs the experts came from the OS page cache (the decode stats show 0 MB read from the files), so this only shows that the default path costs nothing; the disk-bound case is the next item. The spread inside one arm is far larger than any difference between the arms. Base on experts.bin in series 1: 27.21, 35.68, 15.99. Expert-cache hits were the same in every run (65,247 of 122,400).
- Disk-bound A/B (the case where the experts do not fit in RAM and VRAM): 8 alternating runs, order B C C B B C C B (B = experts.bin, C = the same pack with experts.wex4), both packs on the same NVMe SSD, 15-17 GB of RAM free, 999-token prompt + 256 greedy tokens, `STRATA_UNBUFFERED_LOAD=1 --mmap-experts --resident-budget-gib 2 --expert-cache 1000` (1286 VRAM slots, 2.48 GiB) `--spec 2 --prefill 512`. After the RAM copy the mapped view is closed and every expert outside the 2 GiB page-locked RAM copy and the VRAM cache is read with `FILE_FLAG_NO_BUFFERING`, so the OS file cache serves none of these reads. In both arms each run read 643 MB/token from disk during decode (about 86.2k expert blobs per run, 6.6k from RAM, VRAM hit rate 63.1 %), reading took 135-140 ms of each ~155 ms token at about 4.7 GB/s, and each run read 204.9 GB in total. Medians:

  | metric | experts.bin | WEX4 | delta | spread experts.bin / WEX4 |
  | --- | --- | --- | --- | --- |
  | decode tok/s | 6.45 | 6.38 | -1.1 % | 1.5 % / 2.9 % |
  | prefill tok/s | 168.1 | 167.4 | -0.4 % | 0.9 % / 2.4 % |

  The decode delta is not significant (exact permutation test p = 0.11) and is -0.75 % over the 6 runs with identical token streams. Decode bytes and decode read operations are identical between the arms; WEX4 issues about 1.3k more read operations, all at start-up. The only separable difference is +0.8 % reading time per token (about 1 ms), which can equally come from where the two files sit on the disk. 6 of 8 runs gave identical tokens; one run in each arm diverges at the same token, so not because of the format. So with disk-bound decode WEX4 neither gains nor costs measurably: the records hold the same bytes.
- Start-up cost: the header and index checks took 28 to 39 ms (14 starts). File size: 25,179,885,568 B vs 25,146,163,200 B for `experts.bin` (+0.13 %: the 5.7 MB index and the record padding).
- Through the server: `serve/server.py --engine strata --config <run config>` with `--resident-experts --spec 4 --mtp <dir> --max-context 524288 --kv k8v4` in the config's `args`, once with `--experts-wex4 ... --experts-wex4-root ...` on the WEX4 pack and once on the `experts.bin` pack: both loaded in about 20 s (`header and index checked in 30 ms`, 8.81 GiB of experts page-locked in RAM), and one greedy 96-token chat request gave the same reasoning and reply text on both (148.3 vs 149.1 tok/s, 59 of 80 vs 56 of 81 drafts accepted).
- `python tools/check_vendored.py`: ok, `engine/third_party/wex: 3 files (3 upstream)`.
- `python -m pytest terminal/tools/test_iq_pack.py terminal/tools/test_no_llama_download.py terminal/tests/terminal/test_bundle_imports.py`: 52 passed, 1 skipped.
- `tools/check --only py-tests,py-serve,py-tools` on a second Windows PC (Python 3.14), base and branch side by side: the branch adds no failure. Both trees fail the same 100 py-tests and the same three py-tools tests on that machine; py-serve had 9 failures on base and 3 on the branch, all in server start / vision timing tests, and the one that failed on the branch only (`test_vision_ready_pipe ... wait_for_ever`) passes when rerun. This PR changes no `serve/` file.

**Failed:**
- none observed.

**Not run:**
- HIP build (the change touches `strata_gpu_engine.cmake` and `expert_source.cpp`, which HIP compiles): no AMD card or ROCm here. The new reader and its test compile warning-free with g++ 16.2, but the HIP engine build itself is unverified.
- Metal build (only the source list and the refusal changed): no Mac.
- Linux: the `madvise` / `posix_fadvise` release and the Linux `read_direct` path with a WEX4 stride are compiled only on Linux and were not built or run here.
- The desktop app and `strata launch` passing these flags through `user_args` (only a hand-written run config through `serve/server.py` was tried).

## Limits

- File tier only (`--mmap-experts` or `--resident-experts`); the default resident arena does not read WEX4 and the engine refuses the flag there.
- RAW records only. WEX4's compressed codec (WNR3) codes MXFP4 experts, which no Strata pack uses, so it is refused by name.
- Not for NVFP4 (the scale tail is not a GGUF tensor) or Gyro (types 144-146 do not fit the one-byte dtype code) packs; `iq_pack.py` and the engine refuse both.
- No disk saving: the container is 0.13 % larger than `experts.bin`.
- No hashing at run time: an expert is not re-hashed when read; a damaged expert inside the file is found by `wex4_check.py`, not by the engine at start.
- Measured on one rig and one model.

Disclosure: WEX4 and `wex4.py` come from my own Apache-2.0 project, Wex-WRN2. The code was written with AI assistance; I have reviewed it and can answer for every line.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.