#6615 [Engine] PLE: opt-in NGQ2 n-gram table (2.1 bits a value, 13.4 GB instead of 28.8 GB)

open · @go-deung-eo · 0 Kommentare · Auf GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

## Problem and result

On a 64 GB PC, the bundled IQ4_NL PLE table (26.8 GiB) can never be held in RAM: the residency policy needs 26.8 + 31.6 GiB expert arena + 8 GiB headroom = 66.4 GiB. So every token and prompt waits on SSD reads for its n-gram rows. On larger PCs the 26.8 GiB lock competes with everything else for RAM (#5220, #6082).

NGQ2 is a 2-bit trellis format for this table, from our paper (arXiv 2610.21810), published as `dnotitia/Qwen3.8-Flash-Next-NGQ2`. A row is 42 B instead of 90 B, so the table is 12.5 GiB and needs 52.1 GiB free. That fits on a 64 GB PC.

This PR lets Strata read that table when it is passed with `--ple-gguf`. It is opt-in. With any other table, nothing changes.

On two 64 GB-class PCs, the default residency policy keeps NGQ2 in RAM and leaves IQ4_NL on SSD. The engine log says `--ple-io ram chosen by the residency policy (the table is 12.5 GiB, the expert arena still to fill 31.6 GiB and 52.8 GiB of RAM is available)`.

| | RTX 5090 32 GB, 60 GiB RAM | RTX 6000 Ada 48 GB, 62 GiB RAM |
|---|---|---|
| table kept in RAM | NGQ2 yes, IQ4_NL no | NGQ2 yes, IQ4_NL no |
| host PLE time, 4K prompt | 75 → 32 ms | 57 → 37 ms |
| host PLE time, 16K prompt | 152 → 110 ms | 117 → 80 ms |
| 4K prompt tok/s | 6,500 → 6,889 (+6.0 %) | 3,585 → 3,452 (within the spread) |
| 16K prompt tok/s | 8,860 → 9,079 (+2.5 %) | 4,045 → 4,083 (within the spread) |
| decode tok/s | 161.3 → 160.2 (within the spread) | 128.4 → 130.0 (within the spread) |

- Setup: Q2_0 pack @ed59f92 (the AVX2 native pack on the 6000 Ada host's i5-13400F). Only the PLE table changes between runs.
- Method: `strata_bench`, 3 interleaved rounds, a fresh engine each run, median. Host PLE time is the per-prompt `PrefillStats::ms_ple` from `STRATA_PREFILL_TIMING`.
- How much the RAM table shows up end to end depends on the SSD. On the 6000 Ada host direct reads are already fast (probe about 200k rows/s), so the saved 20–40 ms is 1–2 % of a prompt.

Quality is lower than IQ4_NL. We measured it with `quant_quality.py` v1 (4,995 positions), using the BF16 table as the reference:

| table | top-1 | mean KL | KL p99 | PPL (BF16 4.505) |
|---|---|---|---|---|
| IQ4_NL (bundled) | 95.68 % | 0.0193 | 0.237 | 4.464 |
| NGQ2 | 94.47 % | 0.0349 | 0.594 | 4.479 |
| all gains zero (no table) | 85.21 % | 0.2002 | 2.785 | 4.595 |

Row error is relative L2 against BF16 over 4,000 random rows (seed 0):

| format | row error |
|---|---|
| Q8_0 | 0.53 % |
| FP8 | 2.66 % |
| IQ4_NL | 7.59 % |
| NGQ2 | 26.8 % |
| RTN 2-bit, for scale | about 46 % |

The Q8_0 and FP8 values match the figures you reported, so the method is the same.

## Related discussion

- #6192: the residency policy counts every expert blob even when the GPU holds them, so the table stays on NVMe on machines that could keep it. A table half the size fits under either the current or a fixed policy.
- #5220, #6082: when the 26.8 GiB table is admitted, the lock competes with the conversation cache and the prompt chunk.
- #5248, #5749: users already want to pick the n-gram table by RAM budget. NGQ2 adds a smaller option.

## Approach

All changes are host-side. No GPU file and no shared reader are touched.

- `ngram.hpp` / `ngram.cpp`: one more `PleFormat` entry. NGQ2 is marked like NVFP4 (I8 bytes plus `strata.ple.format = "ngq2"`). The row decoder reproduces the checkpoint's own decoder bit for bit.
- The 512 x 2 codebook is a second tensor in the table file (`strata.ple.codebook`), so `gguf_reader.hpp` stays unchanged. With two tensors, `PleTable::open` checks the exact layout for NGQ2, as a single-tensor table file already gets.
- `ple_reader_test.cpp`: NGQ2 joins the all-formats round trip, plus a golden row checked against an independent numpy decoder.
- `terminal/tools/ple_ngq2_pack.py`: turns the published checkpoint into that GGUF. It copies bytes and decodes nothing.

## Validation

**Passed:**
- Revision: this branch on main c6f6427d9 (`ple_reader_test --selftest` rerun there). The builds and checks below ran on the same patch on main 58ee322772; the upstream changes since then do not touch the NGQ2 lines.
- `ple_reader_test --selftest`: all formats, including NGQ2 and its golden row.
  - CUDA build, RTX 5090 (sm_120) host.
  - CUDA build on an RTX 6000 Ada host (sm_89).
  - HIP (`rocm/dev-ubuntu-24.04:7.2.4-complete`, `--preset hip-compile`).
  - SYCL (oneAPI 2026.1.1, the `ple_reader_test` target).
  - Metal: macOS 26.6, M4 Pro, metal-cpp 26.
- Builds of `strata`: CUDA, HIP, SYCL and Metal. Each warning list is identical to main's, and the NGQ2 lines add no warnings.
- Real table (`ple_ngq2_pack.py` output from the HF checkpoint):
  - 4,000 random rows are bit-identical to a numpy reference decoder through direct, mmap and `gather_batch`.
  - Packing from the HF files gives the same table and codebook bytes as the table used for all measurements.
- Default path:
  - IQ4_NL rows (4,000) are byte-identical between main and this branch.
  - The quant_quality TSV is byte-identical between main and this branch on the RTX 6000 Ada (sm_89, Q2_0 AVX2 native pack).
- Quality numbers above:
  - Measured on c7757280 with the same patch.
  - After the rebase, the NGQ2 TSV on the 6000 Ada is byte-identical to the one before it.

**Failed:**
- none observed

**Not run:**
- Engine runs on AMD, Intel or Apple GPUs (HIP, SYCL and Metal were built and selftested only).
- The split engine: it supports IQ4_NL and Q8_0 tables only and will refuse NGQ2.
- Speed after the rebase: the speed numbers are from c7757280.
- An FP8-table quality arm.

## Limits

- One NGQ2 codebook per process. A second NGQ2 table with a different codebook is refused with an error. Qwen3.8-Flash-Next has one table.
- The speed gain comes only from residency. With both tables on `--ple-io direct`, NGQ2 is 1.6–2.1 % slower on prompts. SSD reads per token are the same, about 10.7 (one 4 KiB page per row).
- NGQ2 is admitted only with about 52 GiB of free RAM (12.5 GiB table + 31.6 GiB expert arena + 8 GiB headroom). With less, both tables stay on direct reads and perform alike.
- On both hosts the table was loaded but not locked, because RLIMIT_MEMLOCK allowed 7.8 GiB (#4671).
- No `setup.py` entry. The table is used only when passed with `--ple-gguf`.

AI assistance: Claude Code helped write the patch, run the measurements and draft this text. I reviewed every change and the numbers, and the raw logs are available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01NmF6vwKSTFhomDj7LJsonm

Mehr auf der Site

Links zu Install, Modellen, Releases.