#6077 [Engine] PLE: a VRAM hot tier for n-gram table rows, and stop pinning what is already resident

open · @archadrian · 1 commentaires · Sur GitHub

Server & APINVIDIA / CUDAModels & quantsLinux

Description

### Area
[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

Make the n-gram table cheap to run by putting its hot rows in VRAM instead of forcing the whole table into host RAM.

The problem is a cost mismatch. The PLE table is the largest single object in engram models: 26.8 GiB at IQ4_NL, 32.8 GiB at Q5_0 on our Groot-65B-A6B pack (A6B active, sparse MoE; the expert pool is only 14.9 GiB). Today the only way to keep it hot is host RAM, and 64 GB of RAM is expensive right now, while an extra 4 or 8 GB of VRAM on a 12 or 16 GB card, or an older 24 GB card, often costs nothing at the same market price. The expensive axis is RAM; the cheap axis is VRAM that already sits on the card.

What we measured (our numbers, this week, engine 0.1.54 / build-75, Linux, RTX 3070 Ti and RTX 3090 lanes):

- The table's working set is tiny. We replayed the engine's own row math (`ngram_rows()`, oracle-verified: 23,968 of 23,968 row ids identical against the C++ function) over a 1M-token mixed prose+code session using the serve layer's own tokenizer: 16M lookups touched 3.92M distinct rows, 1.2% of the table, 337 MiB of table bytes. Growth is sublinear (50K tokens: 27 MiB; 1M tokens: 337 MiB).
- LRU stack distance: p50 is 0.14 GiB of rows, p99 is 0.32 GiB. A 1 to 2 GiB VRAM slice covers a full session's working set with 4 to 7x headroom.
- 24.5% of lookups are cold first-touches, mostly order-3 trigram rows. No cache helps those; each is one 90 byte random read, about 9 MB/min at 50K tokens per minute, trivial for NVMe.
- The table's rows are already a runtime parameter (`row_bytes`), and the expert hot tier already does routing-aware VRAM placement. The ask is the same mechanism pointed at n-gram rows: keep the hot rows in VRAM, keep the full table as an evictable cold store in RAM or on disk.
- Second half of the same design: stop pinning what is already resident. On a 24 GB card the expert pool fits fully in VRAM (we ran Groot Gyro-M fully resident on one 3090: 15.3 GiB experts hot, 120+ t/s), yet the host arena stays pinned at 14.9 GiB. And on our 48 GiB test machine the table's `mlock` fails at the default `ulimit -l` ("its pages stay faulted in but may be reclaimed"), so on a tight machine "hot" is a state that can be lost, not a guarantee.

Who needs it: anyone running engram models on 32 or 48 GB machines. With a PLE VRAM tier the host RAM floor for our pack drops from about 46 GiB (pinned arena 14.9 + table 26.8 + dense 1.4 + OS) to about 5 GiB, which turns the 48 GB frame from "kill everything else and hope" into a comfortable run, and reopens the 32 GB frame from the other side: the table stops being a residency requirement at all. Related threads we read: #3357 and #5220 fix how the RAM decision is made; this is the other half, using VRAM that is already on the card. #4105 makes the table load fast; a VRAM tier makes it small in RAM.

### What do you do today instead?

Two workarounds, both worse:
- Run on 64 GB+ machines and take the RAM hit (`--ple-io ram`), which prices the model out of the mass-machine tier.
- Keep the table cold and stream it: the community floor for engram models of this class is 48 GB with the n-gram table left on disk. We also measured a single-variable PLE requant (Q5_0 to IQ4_NL, the engine's own format): pool drops 52 to about 46 GiB at zero decode cost (56.4 t/s vs 57.4 t/s warm at 20K, same protocol, same card), but 46 GiB still does not fit a 48 GB machine with anything else running.

If useful, we can attach the full row-frequency replay (script plus rows banked) or take this on as a PR.

Sur le site

Liens install, modèles, releases.