#6305 [Engine] fingerprint the resolved MTP files, so the SSD conversation tier stays on for GGUF and rt drafters

closed · @mnott · 0 评论 · 去 GitHub 看

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants

说明

Fixes #6281

## Problem and result
Since 0.2.16 the conversation cache has a default-on SSD tier. Before the first spill the engine fingerprints the model, so a spilled conversation is only ever restored by the model and configuration that wrote it. The fingerprint list assumed that `--mtp` always names an MTP pack folder: it asked for `<mtp>/dense.bin`, `dense.txt` and `experts.bin`. Two real drafter layouts have none of those:

- `--mtp <file>.gguf`: the GGUF is the whole drafter.
- `--mtp <rt dir>`: an rt directory from `tools/mtp_rt.py` that holds only `draft_vocab.bin`. The Metal engine (`window.cpp`) then opens the GGUF **beside** it — `STRATA_METAL_MTP_GGUF`, else `<parent>/mtp-q2_0.gguf` — and reads the draft head's token subset from `<gguf dir>/rt/draft_vocab.bin`. This is a common Metal setup (`--mtp .../Strata-data/mtp/rt`, weights in `.../Strata-data/mtp/mtp-q2_0.gguf`):

    strata serve: conversation cache: no SSD tier from now on: the model cannot be fingerprinted (session fingerprint: mtp <dir>/rt/dense.bin: No such file or directory)

The tier then closes for the rest of the run. Disk sessions (`--session`) hit the same list.

After this change the fingerprint lists exactly what the loaders resolve and open: the GGUF a `--mtp` file names, the sibling GGUF an rt directory resolves to (plus its `rt/draft_vocab.bin`), a pack folder's own `dense.txt`/`dense.bin`/`experts.bin` only when they are there, and no entry at all when nothing is opened or the drafter was cleared (`--draft-fallback none`). The SSD tier stays on.

## Related discussion
Issue #6281.

## Approach
- `session_model_inputs` no longer adds an mtp entry: a directory without `dense.bin` is not a missing pack file, and one place must resolve the drafter.
- `session_mtp_pack_files(mtp, draft_vocab, mtp_gguf)` resolves like the loaders (same paths as `mtp.cpp` and Metal `window.cpp`, no second resolution logic):
  - a regular file: that GGUF (role `mtp gguf`), plus `<gguf dir>/rt/draft_vocab.bin` when it exists;
  - a directory with `dense.bin` or `experts.bin`: the pack entries as before (`dense.txt`, the weights present, its own `draft_vocab.bin` when there);
  - an rt directory: `mtp_gguf` if given (generate.cpp passes `STRATA_METAL_MTP_GGUF`), else `<parent>/mtp-q2_0.gguf`, plus `<gguf dir>/rt/draft_vocab.bin` when it exists;
  - a directory that opens nothing, or an empty `--mtp` (a cleared drafter): no entry — nothing is opened for it and a name that resolves to no drafter must not kill the fingerprint.
  - `--mtp-draft-vocab FILE`, when given, replaces the vocabulary entry in every shape.
- The CUDA/HIP/Metal and SYCL `generate.cpp` pass `STRATA_METAL_MTP_GGUF` through, like `window.cpp` reads it.
- Nothing is relaxed: every file the engine loaded stays in the hash (size, head and tail samples) under its role, so another drafter, a different pack or a changed GGUF still yields another fingerprint and its spill is not restored. For real pack folders the list and roles are as before, so existing session files stay valid.
- No placeholder files are needed or wanted.

## Validation
**Passed:**
- `g++ -std=c++20 -Iengine/include engine/src/core/conversation_file_test.cpp engine/src/core/conversation_file.cpp && ./a.out` on v0.2.16 plus this change (macOS arm64): 209 checks passed, including new ones: an rt directory holding only `draft_vocab.bin` fingerprints through the sibling GGUF and is stable across resolutions; editing that GGUF changes the fingerprint; a pack folder lists its own files and no sibling GGUF; an empty folder and a cleared drafter list nothing; `--mtp FILE` is the GGUF with the rt vocab beside it; `--mtp-draft-vocab` and the env override name the right files.
- A probe against the live machine's layout (`--mtp /Users/<user>/Strata-data/mtp/rt`, dir holds only `draft_vocab.bin`) resolves to `/Users/<user>/Strata-data/mtp/mtp-q2_0.gguf` (role `mtp gguf`) and `/Users/<user>/Strata-data/mtp/rt/draft_vocab.bin` (role `mtp draft_vocab.bin`) — the same files the running engine logs it loaded — and fingerprints ok where before it failed with `.../rt/dense.bin: No such file or directory`.
- `cmake -S engine/metal ... && cmake --build ... --target strata` (Metal, macOS arm64): builds and links, no errors.
- Live serve run on Metal (v0.2.16 plus this change, Gyro-L with an rt drafter, `strata serve` with the default SSD tier, real multi-conversation load): start logs `conversation cache: SSD tier in ..., up to 32768 MiB`; under load the cache evicts a conversation and logs `conversation cache: spilled 91040 tokens to the SSD tier (evicted from RAM, written in the background) in 1958.1 ms; on SSD 1, 2115 MiB`; the session file is on disk and no "cannot be fingerprinted" / "no SSD tier from now on" line appears. Before the change the same setup logged that line on its first spill and ran without the tier.

**Failed:**
none observed

**Not run:**
- CUDA, HIP and SYCL builds (the SYCL edit mirrors the CUDA one line for line).
- A restore from the SSD tier was not yet observed in the live run (it needs the spilled conversation to come back); the fingerprint is the same function on write and read, and the unit test checks it is stable.

## Limits
- As before, an edit in the middle of a model file that keeps its size is not detected (head and tail are sampled).
- The rt-directory shape assumes the Metal loader's default GGUF name (`mtp-q2_0.gguf` beside the rt dir) unless `STRATA_METAL_MTP_GGUF` names another; a CUDA build pointed at an rt directory without pack files would load nothing anyway.

本站相关内容

相关页面的快捷入口。