#6615 [Engine] PLE: opt-in NGQ2 n-gram table (2.1 bits a value, 13.4 GB instead of 28.8 GB)
open · @go-deung-eo · 0 comentarios · En GitHub
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quants
Descripción
## Problem and result On a 64 GB PC, the bundled IQ4_NL PLE table (26.8 GiB) can never be held in RAM: the residency policy needs 26.8 + 31.6 GiB expert arena + 8 GiB headroom = 66.4 GiB. So every token and prompt waits on SSD reads for its n-gram rows. On larger PCs the 26.8 GiB lock competes with everything else for RAM (#5220, #6082). NGQ2 is a 2-bit trellis format for this table, from our paper (arXiv 2610.21810), published as `dnotitia/Qwen3.8-Flash-Next-NGQ2`. A row is 42 B instead of 90 B, so the table is 12.5 GiB and needs 52.1 GiB free. That fits on a 64 GB PC. This PR lets Strata read that table when it is passed with `--ple-gguf`. It is opt-in. With any other table, nothing changes. On two 64 GB-class PCs, the default residency policy keeps NGQ2 in RAM and leaves IQ4_NL on SSD. The engine log says `--ple-io ram chosen by the residency policy (the table is 12.5 GiB, the expert arena still to fill 31.6 GiB and 52.8 GiB of RAM is available)`. | | RTX 5090 32 GB, 60 GiB RAM | RTX 6000 Ada 48 GB, 62 GiB RAM | |---|---|---| | table kept in RAM | NGQ2 yes, IQ4_NL no | NGQ2 yes, IQ4_NL no | | host PLE time, 4K prompt | 75 → 32 ms | 57 → 37 ms | | host PLE time, 16K prompt | 152 → 110 ms | 117 → 80 ms | | 4K prompt tok/s | 6,500 → 6,889 (+6.0 %) | 3,585 → 3,452 (within the spread) | | 16K prompt tok/s | 8,860 → 9,079 (+2.5 %) | 4,045 → 4,083 (within the spread) | | decode tok/s | 161.3 → 160.2 (within the spread) | 128.4 → 130.0 (within the spread) | - Setup: Q2_0 pack @ed59f92 (the AVX2 native pack on the 6000 Ada host's i5-13400F). Only the PLE table changes between runs. - Method: `strata_bench`, 3 interleaved rounds, a fresh engine each run, median. Host PLE time is the per-prompt `PrefillStats::ms_ple` from `STRATA_PREFILL_TIMING`. - How much the RAM table shows up end to end depends on the SSD. On the 6000 Ada host direct reads are already fast (probe about 200k rows/s), so the saved 20–40 ms is 1–2 % of a prompt. Quality is lower than IQ4_NL. We measured it with `quant_quality.py` v1 (4,995 positions), using the BF16 table as the reference: | table | top-1 | mean KL | KL p99 | PPL (BF16 4.505) | |---|---|---|---|---| | IQ4_NL (bundled) | 95.68 % | 0.0193 | 0.237 | 4.464 | | NGQ2 | 94.47 % | 0.0349 | 0.594 | 4.479 | | all gains zero (no table) | 85.21 % | 0.2002 | 2.785 | 4.595 | Row error is relative L2 against BF16 over 4,000 random rows (seed 0): | format | row error | |---|---| | Q8_0 | 0.53 % | | FP8 | 2.66 % | | IQ4_NL | 7.59 % | | NGQ2 | 26.8 % | | RTN 2-bit, for scale | about 46 % | The Q8_0 and FP8 values match the figures you reported, so the method is the same. ## Related discussion - #6192: the residency policy counts every expert blob even when the GPU holds them, so the table stays on NVMe on machines that could keep it. A table half the size fits under either the current or a fixed policy. - #5220, #6082: when the 26.8 GiB table is admitted, the lock competes with the conversation cache and the prompt chunk. - #5248, #5749: users already want to pick the n-gram table by RAM budget. NGQ2 adds a smaller option. ## Approach All changes are host-side. No GPU file and no shared reader are touched. - `ngram.hpp` / `ngram.cpp`: one more `PleFormat` entry. NGQ2 is marked like NVFP4 (I8 bytes plus `strata.ple.format = "ngq2"`). The row decoder reproduces the checkpoint's own decoder bit for bit. - The 512 x 2 codebook is a second tensor in the table file (`strata.ple.codebook`), so `gguf_reader.hpp` stays unchanged. With two tensors, `PleTable::open` checks the exact layout for NGQ2, as a single-tensor table file already gets. - `ple_reader_test.cpp`: NGQ2 joins the all-formats round trip, plus a golden row checked against an independent numpy decoder. - `terminal/tools/ple_ngq2_pack.py`: turns the published checkpoint into that GGUF. It copies bytes and decodes nothing. ## Validation **Passed:** - Revision: this branch on main c6f6427d9 (`ple_reader_test --selftest` rerun there). The builds and checks below ran on the same patch on main 58ee322772; the upstream changes since then do not touch the NGQ2 lines. - `ple_reader_test --selftest`: all formats, including NGQ2 and its golden row. - CUDA build, RTX 5090 (sm_120) host. - CUDA build on an RTX 6000 Ada host (sm_89). - HIP (`rocm/dev-ubuntu-24.04:7.2.4-complete`, `--preset hip-compile`). - SYCL (oneAPI 2026.1.1, the `ple_reader_test` target). - Metal: macOS 26.6, M4 Pro, metal-cpp 26. - Builds of `strata`: CUDA, HIP, SYCL and Metal. Each warning list is identical to main's, and the NGQ2 lines add no warnings. - Real table (`ple_ngq2_pack.py` output from the HF checkpoint): - 4,000 random rows are bit-identical to a numpy reference decoder through direct, mmap and `gather_batch`. - Packing from the HF files gives the same table and codebook bytes as the table used for all measurements. - Default path: - IQ4_NL rows (4,000) are byte-identical between main and this branch. - The quant_quality TSV is byte-identical between main and this branch on the RTX 6000 Ada (sm_89, Q2_0 AVX2 native pack). - Quality numbers above: - Measured on c7757280 with the same patch. - After the rebase, the NGQ2 TSV on the 6000 Ada is byte-identical to the one before it. **Failed:** - none observed **Not run:** - Engine runs on AMD, Intel or Apple GPUs (HIP, SYCL and Metal were built and selftested only). - The split engine: it supports IQ4_NL and Q8_0 tables only and will refuse NGQ2. - Speed after the rebase: the speed numbers are from c7757280. - An FP8-table quality arm. ## Limits - One NGQ2 codebook per process. A second NGQ2 table with a different codebook is refused with an error. Qwen3.8-Flash-Next has one table. - The speed gain comes only from residency. With both tables on `--ple-io direct`, NGQ2 is 1.6–2.1 % slower on prompts. SSD reads per token are the same, about 10.7 (one 4 KiB page per row). - NGQ2 is admitted only with about 52 GiB of free RAM (12.5 GiB table + 31.6 GiB expert arena + 8 GiB headroom). With less, both tables stay on direct reads and perform alike. - On both hosts the table was loaded but not locked, because RLIMIT_MEMLOCK allowed 7.8 GiB (#4671). - No `setup.py` entry. The table is used only when passed with `--ple-gguf`. AI assistance: Claude Code helped write the patch, run the measurements and draft this text. I reviewed every change and the numbers, and the raw logs are available. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01NmF6vwKSTFhomDj7LJsonm
En el sitio
Enlaces a install, modelos, releases.