#6401 [Engine] oomd_pacer: discover split --native GGUF shards and keep PLE cached without --ple-gguf

open · @larryllr · 0 comentários · No GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

## Problem and result

Commit `12d491d` added `ple_table_file(args)` to `terminal/serve/oomd_pacer.py` so `make_room()` keeps the PLE shard's page cache when the engine maps or locks the table (`--ple-io ram` / `mmap` / default), avoiding a cold re-read of the 26.8 GiB PLE table.

However, for 3+ shard GGUF sets (such as Unsloth `UD-Q4_K_XL`, `UD-IQ4_XS`, `UD-Q5_K_XL`, `UD-Q6_K_XL`, and `UD-Q8_K_XL`), `terminal/setup.py` omits `--ple-gguf` and passes only shard 1 after `--native` (relying on `engine/src/program/generate.cpp` to expand sibling shards via `gguf_split_paths()` and locate `per_layer_token_embd.weight` by tensor name). Because `oomd_pacer.py` only inspected literal command-line paths and `--ple-gguf`:

1. `_arg_files(args)` missed sibling shards (`00002-of-0000N`..`0000N-of-0000N`), undercounting `total` in `make_room()` (notably on Unsloth `UD-Q4_K_XL` / `UD-Q5_K_XL`, where shard 1 is metadata-only) and leaving shards 2..N cached even when `make_room()` triggered.
2. `ple_table_file(args)` returned `None` whenever `--ple-gguf` was omitted (3+ shard configs and single-file `--native` launches without `--ple-gguf`), so the PLE shard's cache was still dropped under `--ple-io ram` / `mmap` / default. It also kept returning the `--ple-gguf` file when `--no-ple` was passed.
3. `trim_files(args)` missed sibling shards and did not exclude the PLE shard, so layouts where shard 1 holds `per_layer_token_embd.weight` (Swift) or single-file GGUFs still had their PLE table trimmed while loading.

After this change:
- `_arg_files(args)`, `ple_table_file(args)`, and `trim_files(args)` expand `<name>-00001-of-0000N.gguf` split siblings for `--native` in split order.
- `ple_table_file(args)` still prefers explicit `--ple-gguf`, returns `None` for `--ple-io direct` or `--no-ple`, and otherwise scans `--native` shard GGUF headers (header-only) for `per_layer_token_embd.weight`.
- `trim_files(args)` trims all `--native` shards except the kept PLE shard.

## Related discussion

none (focused follow-up fix to commit `12d491d` in `terminal/serve/oomd_pacer.py`).

## Approach

- Kept `terminal/serve/oomd_pacer.py` standard-library-only (`struct`, `re`) so minimal server deployments (`terminal-minimal-server`) require no extra dependencies.
- Added `_gguf_split_siblings()` matching `engine/include/strata/artifact/gguf_split.hpp` (`<stem>-NNNNN-of-MMMMM.gguf`, preserving caller directory/symlinked names, skipping missing files without raising).
- Added `_gguf_has_tensor()` to inspect only GGUF v2/v3 headers (short-circuiting immediately when `n_tensors == 0` such as metadata-only shard 1, bounds-checking all counts/lengths, skipping metadata payloads without allocating large vocab arrays, never reading tensor data, never raising).
- In `_arg_files(args)`, preserved explicit arguments and appended only auto-discovered `--native` siblings not already present by `_same_file()` before walking folders, so split shards are never truncated by `ROOM_MAX_FILES` and existing explicit-arg tests keep exact parity.
- Added unit tests in `terminal/tests/terminal/test_oomd_room_ple.py` covering 4-shard metadata-only-shard-1 layouts, `--ple-io ram/mmap/direct`/default, `--no-ple`, Swift shard-1-PLE layout, side-loaded `--ple-gguf` precedence, single-file GGUFs, `trim_files()`, and corrupt/truncated headers.

## Validation

**Passed:**
- Revision: `main` (`3142b66`) + PR commits through `9283d16` on Windows 11, Python 3.12.
- `PYTHONPATH=. python -m unittest discover -s tests/terminal -p "test_oomd*.py" -v` (in `terminal/`): `Ran 61 tests in 0.233s — OK (skipped=5)` (all 52 existing non-Linux-only tests + 9 new test methods in `MultiShardNativeAndAutoPle`).
- `PYTHONPATH=. python -m unittest serve.test_cache_drop serve.test_lastrun serve.test_lifecycle -v` (in `terminal/`): `Ran 23 tests in 5.002s — OK (skipped=11)`.

**Failed:**
- none observed.

**Not run:**
- Linux-only `posix_fadvise` and `/proc` tests in `test_oomd_pacer.py` and `test_oomd_room.py` (skipped on Windows; covered by the Linux CI runners on this PR).
- Live Linux desktop / container load test under `systemd-oomd` (no Linux `systemd-oomd` host available locally).
- Engine C++/CUDA/HIP/SYCL/Metal builds (no engine files were modified).

## Limits

- `_gguf_has_tensor()` parses standard GGUF v2/v3 metadata types (nested arrays up to depth 4); corrupt or unrecognized headers safely return `False` without raising and fall back to existing behavior.
- Split sibling expansion applies to `--native` paths matching `-NNNNN-of-MMMMM.gguf` with `2 <= total <= 1024`.

No site

Links install, modelos, releases.