#2797 [Linux, RX 7900 XTX gfx1100, 0.1.44.1, ntfs3] --ple-io direct's PLE reads run at ~950 IOPS (3.7 MiB/s) while the same file does 70,406 IOPS from another process; boot probe says "fine", and the #1407 I/O allowance cannot fire for small random reads

本文

**Strata version, GPU, OS:** `0.1.44.1` · AMD Radeon RX 7900 XTX 24 GB (gfx1100, ROCm 7.2.3,
HIP) · Linux 7.0.0-38-generic, Intel B760M D5 (no AVX-512), 20 CPUs, 62.6 GB RAM

**Summary.** On Linux + HIP (RX 7900 XTX, gfx1100, engine 0.1.44.1, stock config) the default
`--ple-io direct` PLE read path delivers **~950 random 4 KiB reads/s (3.7 MiB/s)** while a
prompt chunk is read, so layer 1 of one chunk never finishes inside the 60 s watchdog and the
engine aborts itself (exit -6, `server.py` restarts it, clients see 503). Five occurrences in
two engine sessions.

**The drive is not the limit, and it is not the Windows unbuffered path.** The model GGUF is on a
non-rotational NVMe, and a standalone process reading the *same file* with the *same* random
4 KiB `O_DIRECT` pattern reaches **70,406 IOPS / 275 MiB/s / 0.22 ms**, measured at the block layer
(`/sys/block/nvme0n1/stat`), not in userspace. That run overlapped the engine's stalled window:
while the engine's 16 pool threads sat in state `D` and `/proc/<pid>/io` advanced at ~3.7 MiB/s, a
second process doing identical reads on the same file still got 45–91k IOPS.

**The startup probe gives a false all-clear.** On this machine the boot probe prints

```
strata generate: PLE direct read probe: 34667 rows/s (fine)
```

(above the 15,000 rows/s bar in `generate.cpp:3314`), so no warning is printed. But that is ~36x
what the same process, same `PleReader`, same `DirectFile` pool delivers later at runtime. The
probe measures 4096 rows in ~0.12 s at boot, before the GPU work and the CPU expert pool exist; it
does not predict the prompt path.

## Environment

- **Engine:** 0.1.44.1, HIP, ROCm 7.2.3, stock config (no `STRATA_*` env set)
- **GPU:** AMD Radeon RX 7900 XTX (gfx1100), 24 GB
- **Host:** Linux 7.0.0-38-generic, Intel B760M D5 (no AVX-512), 20 CPUs, 62.6 GiB RAM, 0 MiB swap
- **Disk:** ZHITAI Ti600 1 TB NVMe on `nvme0n1p4`, **NTFS mounted with `ntfs3`**, rotational=0,
  the 44.9 GB GGUF is heavily fragmented (`filefrag` shows many extents)
- **Model:** Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S, PLE table **320,001,536 rows x 90 B = 28.8 GB**,
  `--ple-gguf` pointing at the same shard as `--native`
- **Config:** `--prefill auto --spec 4 --mtp ... --max-context 262144 --kv q4_0 --vision
  --vram-reserve-mib 1024 --pool-workers 9 --conversation-cache-mib 4096 ...`
- `--ple-io` left at the default (`direct`); `STRATA_IO_THREADS` unset

## What the engine logs

```
strata serve: no progress for 60 s during a request (reading the prompt (batched): layer 1 of the
prompt chunk from token 8104) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.44.1): stage "reading the prompt (batched): layer 1 of the
prompt chunk from token 8104" for 60 s; 0 layers served since the last finished step
  VRAM free when this request began: 912 MiB
  expert pool: epoch 0, batch epoch 0: 0 of 0 jobs claimed, 0 done; 9 of 9 workers parked, 9 sleeping
  expert pool threads: w0..w8=sleeping; host idle for 192230 ms
  verify window: 0 tokens at position 0, host at layer step 1; the GPU rang 0
  threads waiting on the disk (state D): 16 of 37 (blk_io_schedule x16)
  memory: 49390 MiB resident, 0 MiB in swap, 8768 MiB RAM available; 7 major page faults so far
```

Stages seen across the five stalls: `layer 1 of the prompt chunk from token 0`, `... from token 8104`,
and one `waiting for the GPU (attention, router) at layer 1`. VRAM was never low (743–912 MiB free),
so this is not the #2065 card-full class. `dmesg` has no I/O errors, no timeouts, no hung-task
warnings; `nvme_core.io_timeout` is 30.

## Measurements (2026-10-17, same day, same machine, same file)

| what | random 4 KiB O_DIRECT | notes |
| --- | --- | --- |
| standalone process, 16 threads, 10 s | **70,406 IOPS, 275 MiB/s, 0.22 ms** | block-layer counters, `/sys/block/nvme0n1/stat` |
| standalone process, 16 threads, 15 s | 67,545 IOPS, 264 MiB/s, 278 ms max | userspace timing |
| standalone process, 64 threads | 75,269 IOPS, 294 MiB/s, 5.7 ms max | userspace timing |
| the engine's boot probe | 34,667 rows/s ("fine") | `ple_direct_rows_per_s`, `generate.cpp:1778` |
| **the engine, during the stall** | **~950 IOPS, 3.7 MiB/s, ~16.8 ms/read** | `/proc/<pid>/io` sampled every second; 16 threads in D |

The per-read latency is ~76x worse inside the engine than from a standalone process on the same file.

## Three things I think are wrong, independent of the root cause

1. **The #1407 I/O allowance can never engage for this read pattern.** `generate.cpp:9640` treats the
   file tier as busy only when `majflt + (read_bytes >> 20)` grows by **>= 16 in one second**. A PLE
   gather is a large number of *small* random 4 KiB reads: at ~950 reads/s that is ~4 MiB/s, and the
   whole run had **7 major page faults**. So the allowance (up to `STRATA_WATCHDOG_IO_S`, default 600 s)
   never starts and the engine is *guaranteed* to abort at 60 s even when it is making steady I/O
   progress. #1425 asked "should an I/O-bound prompt read be able to trip the 60 s no-progress
   watchdog?" and this looks like the answer is: it cannot be helped today, by construction. A read
   counter (reads/s or reads outstanding) would catch this shape; MiB/s does not.

2. **`STRATA_IO_THREADS` is capped at 16 on Linux but `ple_inflight` defaults to 256.** On POSIX each
   pool thread does one blocking `pread`, so the thread count *is* the queue depth
   (`direct_file.cpp:319`, `io_threads(16)`), while `generate.cpp:402` defaults `ple_inflight` to 256
   and comments that 64 "left the SSD half idle". So 256 jobs are queued and only 16 can ever reach
   the drive. `docs/INTEL.md` already documents 16 -> 64; the default still disagrees with the
   comment that justifies it.

3. **Neither documented mitigation is usable on a 64 GB machine running IQ3_S.** `--ple-io mmap`
   needs page cache for a 28.8 GB table and `--ple-io ram` needs 28.8 GB of RAM, but the expert arena
   is already 46.84 GiB resident (and pinned via `cudaHostRegister`), leaving **8768 MiB** available.
   So the log's advice (`#605: --ple-io ram`, `--ple-io mmap`) cannot be followed, and there is no
   supported workaround for this configuration.

## Questions

1. Is the ~76x per-read gap between the engine's runtime path and the boot probe on the same pool
   expected, or is something in the request path (the `aux_cpus` pinning that switches on per request,
   or `--native` mmap and `--ple-io direct` sharing one GGUF file on `ntfs3`) serialising the reads?
2. Would a read-count based file-tier signal (rather than MiB/s + major faults) be the right fix for
   the watchdog allowance?
3. Does anyone have a measurement of `--ple-io direct` on `ntfs3` specifically? All the earlier
   reports I could find (#1425, #1629, #2040, #1407) are Windows, ext4, or loop devices.

## What I did not test

- I did not move the model to an ext4 filesystem, so I cannot separate `ntfs3` from the engine's own
  read path. That is the cheapest next test and I can run it.
- I did not sample `/sys/block/nvme0n1/stat` *during* a stall. That single measurement splits the two
  remaining hypotheses: device IOPS ~950 means the requests reach the drive and are slow there;
  device IOPS ~0 with 16 threads still in `D` means they are stuck above the drive. I can reproduce
  and capture this, plus `gdb -p <pid>` backtraces for the 16 blocked threads.
- Three older stalls in my log are a different class (0 threads in state D, VRAM free 448-450 MiB,
  #2065); I am not claiming them here.

## Related

#1425 (closed for 0.1.42; the maintainer invited a 0.1.44 re-report), #1629, #1407, #2040, #1984,
#605.