Pull requests / #79

#79 linux: make DirectFile reads actually asynchronous

closed · @andrewcoul · 0 comentarios · En GitHub

BenchmarksNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

## What

On Linux, `DirectFile::submit()` ran a blocking `pread` and queued the completion, so the PLE reader's `max_inflight` window never had more than one read in flight: the reads ran back to back on the caller's thread. The Windows backend uses overlapped I/O, so it was already asynchronous there.

`submit()` now queues the request and a pool of threads (`STRATA_PLE_IO_THREADS`, default 32) issues the preads. `wait()` and `wake()` keep their contract (`wake()` shows up as one `WAKE_TAG` completion, `wait(timeout)` blocks up to the timeout). I removed the "Phase L replaces this with io_uring" comment; if io_uring is still planned, this is a stepping stone toward it.

## Measurements (one machine)

Ryzen 7 7800X3D, RTX 4080 SUPER, 32 GB RAM, NVMe, Linux, IQ3_XXS pack. These were measured together with the tiered-experts work (the follow-up PR), where cold experts also stream from the SSD:

- Decode, 400 tokens: time blocked in the PLE reader 640-730 ms -> 65-83 ms; read latency p50 ~2.2 ms -> ~0.2 ms.
- Prefill of a 2.3K-token prompt: the PLE block ~1.8 s -> ~0.1 s.
- End-to-end decode tok/s on its own was within noise (49.6 / 49.1 -> 50.9 / 49.7). It matters more when other work competes for the same time. On a machine where the table is warm in RAM I'd expect little difference.

## Tests

`ple_reader_test --selftest` passes: 3 plain runs, 1 under ThreadSanitizer (no reports), and with `STRATA_PLE_IO_THREADS=1` and `=128`. I built it by hand (`g++` on `ple_reader_test.cpp`, `ple_reader.cpp`, `kernels/ngram.cpp`, `platform/direct_file.cpp`) because `-DSTRATA_BUILD_TESTS=ON` does not configure on `main` (the `tests/` directory and several `bench/micro` sources are not in the repo). Not run on Windows; that code is unchanged.

## Open

The thread count default (32) is a guess that worked here; happy to tie it to `max_inflight` or change it.

En el sitio

Enlaces a install, modelos, releases.