Pull requests / #2073
#2073 PLE: --ple-io direct stays direct when the start-up probe measures the disk slow (STRATA_PLE_AUTO_MMAP=1 opts into the mmap switch); PLE read diagnostics
open · @imanu86 · 0 comments · View on GitHub
Multi-GPUNVIDIA / CUDAModels & quantsWindows
Description
## Title Issue: none (a start-up default that misfired on a slow SSD, and read diagnostics). ## Summary Since 0.1.42 the engine probes the PLE table at start-up and, under 15,000 rows/s, switches `--ple-io direct` to `mmap` on its own. The probe reads 64 rows a batch from a cold table, a far lower queue depth than a prompt chunk's batched read, so a healthy SATA SSD can measure "slow" and the switch costs a long prompt more than three times its time. This change keeps the probe and its warning but makes the switch **opt-in** (`STRATA_PLE_AUTO_MMAP=1`); an explicit `--ple-io` is untouched. It also adds a few PLE read diagnostics. ## Why Measured on one PC (PLE table on a SATA SSD, 131K-token prompt, two-card layer split): the probe read **1,259 rows/s**, switched to mmap, and the prompt took **260 s**; the same engine with direct I/O took **75 s** and decode was 3.9% faster (the mmap mode also drops the row cache and the in-flight reads). mmap faults the rows in one 4 KiB page at a time; direct I/O keeps `--ple-inflight` reads in flight and the chunk read-ahead hides them. The probe's figure is a cold, low-depth read and does not predict the prompt path's throughput. ## What changed - `src/program/generate.cpp`: the probe still runs and prints `PLE direct read probe: N rows/s`; under the threshold it warns and **keeps direct** unless `STRATA_PLE_AUTO_MMAP=1` (then it switches as before). With `STRATA_DECODE_TIMING` the decode summary adds the primary verifier's PLE gather time per window and the reader's cumulative counters before and after the request (`strata decode PLE primary/before/after`). - `src/ngram/ple_reader.cpp`, `include/strata/ngram/ple_reader.hpp`: `ple io:` adds the read mean, the maximum and the count of reads over 100 ms (all reads, where p50/p99 cover the last 65,536). - `src/prefill/prefill.cpp`: a chunk that waited more than 1 s for its PLE rows logs one line (`strata prefill: the chunk at token N waited X ms for its PLE rows`). - `src/kernels/ngram.cpp`, `include/strata/core/verify.hpp`: the gather time counter. No kernel changes; the output tokens are the same in both I/O modes. ## Measurements RTX 2080 Ti 22 GB + RTX 3060 12 GB, Windows 11, CUDA 12.6, `--layer-split 16 --pipeline-windows 2`, Qwen3.8-Flash-Next IQ3_XXS, 131K-token prompt, PLE table on a SATA SSD (cold): | | prompt 131K | decode (131K, 1024 tokens) | |---|---|---| | 0.1.42 default (probe -> mmap) | 260 s | reference | | direct kept (this change) | **75 s** | **+3.9%** | With the table on an NVMe the probe says `fine` (91,220 rows/s here) and nothing changes. Built and started on v0.1.43 (`7695859a`, RTX 2080 Ti + RTX 3060 layer split, Windows): the probe reports `43,290 rows/s (fine)` on the NVMe, the engine serves as before; the same code has run in this fork's daily build since 0.1.42. ## Risks / limits A disk that really is too slow for direct I/O keeps direct unless `STRATA_PLE_AUTO_MMAP=1` is set; the warning says so and names the variable. `--ple-io ram` and an explicit `--ple-io mmap` behave as before. Part of a series from a public patch registry (`stack/` in the fork imanu86/Strata-2080Ti): one lever per PR, each opt-in or bit-exact, measured on the same two-card box.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.