#5588 [Engine] --ple-gguf: a side-loaded PLE table now wins the by-name resolution over the embedded copy

closed · @noahark · 0 Kommentare · Auf GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Beschreibung

## Problem and result

Before: `--ple-gguf PATH` was parsed and its file opened by the PleTable row gather, but the same file was also appended to the native dense shard list. A standalone table file (Unsloth's standalone engram GGUFs) is not a member of the model's split family: its `split.no` collides with one of the model's own shards, the dense loader's metadata validation rejects the collision ("duplicate split shard number"), and the engine died at start. Whatever brought the deployment back up (the watchdog, the operator) restarted it without the flag - so a higher-precision side-loaded engram looked silently ignored: on 3x V100 32GB, layer split 16/16/16, engines 0.1.55 and 0.1.58, UD-IQ4_XS of Qwen3.8-Flash-Next 125B, ten fixed temp-0 probes (~3300 tokens) came back byte-identical with and without the flag.

After: a standalone table file never joins the dense shard list. The table rows are served from `ple_gguf` through the PleTable row gather, which opens exactly this file, so the side-loaded table takes effect, and a stderr line states which file serves the table. Files that ARE shards of the model (the by-name default) keep the old behavior unchanged.

## Related discussion

none yet; targets the use case of #3936.

## Approach

A standalone table file must not join the dense list at all - it is not a shard of the model, so the dense loader's split-family validation can never apply to it. The rows are served from `ple_gguf` through the existing `PleTable::open` path; a stderr line reports the source. No kernel work involved: the PleFormat table already ships Q8_0 / Q5_0 / Q5_1 / Q4_0 / BF16 / NVFP4 row readers and the split-side row reader accepts IQ4_NL and Q8_0. +14/-2.

## Validation

Exact commands, revision and environment: engine built from this branch (ple-gguf-override@672924d, base v0.2.12 / engine 0.1.58), CUDA 12.4 sm_70, Windows 10 19044, 3x Tesla V100 32GB (TCC), driver 581.80. Model: Qwen3.8-Flash-Next 125B UD-IQ4_XS (3 shards), `--layer-split 16,32`, 100% expert residency; side table: the standalone Q8_0 engram GGUF (~50.7 GB, RMSE 0.0053 vs the embedded IQ4_NL's 0.0757).

Serving command shape: `strata.exe --serve --pack <pack> --native <model>-00001-of-00003.gguf --layer-split 16,32 --ple-gguf <standalone-engram-Q8_0.gguf>`. Probes: ten fixed questions at temperature 0 through `/v1/chat/completions`, diffed byte-for-byte between the flagged engine and the embedded-table baseline (watchdog paused so every probe was answered by the flagged engine).

**Passed:**
- Flagged engine starts on the 3-GPU split deployment (no "duplicate split shard number" exit) and logs `PLE table served from --ple-gguf ... (standalone table; the model shards' embedded copy is not used)`; PLE direct-read probe ~31k rows/s on NVMe.
- The same ten probes now diverge from the embedded-table baseline (8/10 answers differ, first divergence at token 0-124, the knowledge-heavy ones earliest) - the flag is no longer silently ignored.
- Decode throughput stays within noise of stock (115-140 t/s samples vs the stock 130.8-130.9 four-band baseline).
- Without the flag the by-name default is unchanged (same probes byte-identical to the pre-fix engine); CI `check` and `metal-link` both pass.
- Running as our daily deployment since (engine 0.1.58 + this patch, 3x V100).

**Failed:**
- none observed.

**Not run:**
- CUDA builds other than our CUDA 12.4 sm_70 Windows one (no sm_80+/sm_90 run), and no HIP, SYCL or Metal runs of this path.
- Side tables other than Q8_0 (Q5_0 / Q5_1 / Q4_0 / BF16 / NVFP4 row readers exist upstream but were not exercised with a standalone file).

## Limits

Only the Q8_0 standalone side table is exercised end-to-end; the split-side row reader accepts IQ4_NL and Q8_0, so other PleFormat tables keep their existing reader coverage but stay unverified with a standalone file on a split deployment.

Mehr auf der Site

Links zu Install, Modellen, Releases.