Pull requests / #3280
#3280 [Engine] --ple-gguf: accept a table-only PLE shard from another file set (Q8_0 table beside IQ3_S / UD-Q4_K_XL)
open · @q8atnight · 0 comentários · No GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux
Descrição
**Problem.** Passing `--ple-gguf` with the PLE table of a *different* file set stops the engine at start-up: ``` strata generate: native dense projections: native dense: additional shard must match the architecture-validated first shard's split metadata ``` The use case: running a smaller quant (ISTA GSQ-RCO IQ3_S, unsloth UD-Q4_K_XL) with the **Q8_0** n-gram/PLE table that unsloth ships as UD-Q5_K_XL / UD-Q6_K_XL / Q8_0 shard 3. That table is read from the SSD, so it costs no RAM. UD-Q5_K_XL shard 3 holds exactly one tensor, `per_layer_token_embd.weight`, and has the same sha256 (`34efd79a…`) as the UD-Q6_K_XL shard 3 listed in docs/UNSLOTH_Q6.md. **Cause.** With `--native`, generate.cpp appends `o.ple_gguf` to `o.native_dense_gguf` unless the file is a `strata-ple` table. A foreign shard then fails native_dense's split-metadata check, even though it holds no projections at all. **Fix (4 lines, src/program/generate.cpp).** A PLE file whose only tensor is `per_layer_token_embd.weight` is treated like a `strata-ple` table: it is used for the table and not added to the dense shards. Nothing else changes. When `--ple-gguf` is not given, or names one of the model's own shards, the code path is the same as before. **Validation (passed).** - Hardware: 2x RTX 3090 (sm_86, layer split 24), Ryzen 9 3950X, 121 GiB RAM, Linux, CUDA 13.0 build. - Base: v0.2.0-beta `3d1fb480` with this commit. - IQ3_S and UD-Q4_K_XL start with `--ple-gguf <UD-Q5_K_XL-00003-of-00006.gguf>`. The log shows `PLE on, table 320001536 rows of …UD-Q5_K_XL-00003-of-00006.gguf`. - Without the commit, the same configs refuse to start with the error above. - Deterministic gate (greedy; 3 prompts + 2 questions on a 20K-token prompt): stock v0.2.0 and v0.2.0 with this commit give **identical output** on IQ3_S, UD-Q4_K_XL and UD-Q5_K_XL. Every model was run with the Q8_0 table in both arms. - Speed, UD-Q4_K_XL int8 KV, Q8_0 table against its own IQ4_NL table: within noise on decode and prefill (flashbench 1K–128K). Full numbers in #3278. **Not run.** HIP and SYCL builds (the change uses only `GgufFile::tensors()`, no backend code). The branch sits on `85b75dcb` (my fork cannot push the newer workflow files). The same commit cherry-picks cleanly onto current main `e88ee39d`. It was built and tested on the v0.2.0-beta base above, not on main. **Limits.** The check is by tensor list: a one-tensor shard that holds only the PLE table. A foreign table shard that also carried projections would still be refused, which is correct. *Written, built and tested by Claude Opus 5.5 working as an agent on this machine.*
No site
Links install, modelos, releases.