Pull requests / #3391

#3391 [Engine] preserve prompt loan residency without token graph

open · draft · @j-luwierski · 0 comentários · No GitHub

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux

Descrição

## Problem and result

On current main (0.2.1, 90767252), a profile-filled IQ3_XXS native pack with --prefill auto and --no-token-graph could not borrow the expert cache. The residency table was only built for the token-graph hit path, so the prompt path tried to allocate its own 8192-token buffers and failed with 23 MiB free while requesting an 80 MiB buffer.

The fix builds the residency table whenever prefill borrowing is available, independently of token-graph capture. On the same RTX 4070 Ti SUPER, the updated build borrowed 2489 cache slots, refilled them after the prompt, processed 12,287 tokens, and generated one token. A token-graph control also completed and produced the same token.

## Related discussion

- Original report and maintainer scope agreement: https://github.com/Niko1221/Strata/pull/1131.
- Parent context: https://github.com/Niko1221/Strata/pull/796.

## Approach

The residency decision covers borrowing availability and a positive prefill chunk. The table is staged for either reader: the token-graph hit path or the prompt loan. Graph hit wiring remains gated by graph capture. A borrowing-only staging failure warns and disables borrowing for that run; graph staging failures retain their existing fatal behavior. Loan/refill trace lines report the slot range and any slots still lent.

## Validation

Tested revision: 70660fa3 on 90767252. Environment: Linux, NVIDIA RTX 4070 Ti SUPER 16 GB, CUDA 13.4.92, CUDA arch 89, Ryzen 9 5900X, 125 GiB RAM. Model: Qwen3.8-Flash-Next IQ3_XXS native pack, int8 KV, 32K context, MTP spec 4. The prompt file contained 12,288 copies of token ID 198.

**Passed:**

- cmake --build build-repro --target strata -j 12 — CUDA Release build passed.
- cmake --build build-repro --target borrow_residency_test -j 12 and ctest --test-dir build-repro -R '^borrow_residency_test$' --output-on-failure — passed.
- Baseline --prefill auto --no-token-graph reproduction on 90767252: exited 1 at prefill initialization with 23 MiB free while the 8192-token chunk requested an 80 MiB buffer.
- Same run on 70660fa3: completed the 12,287-token prompt and generated token 198; the log reported an 8192-token chunk, 2489 borrowed cache slots, and all 2489 slots refilled.
- Token-graph control on 70660fa3: completed the same prompt and generated token 198.
- git diff --check — passed.

Runtime prompt setup: python3 -c 'from pathlib import Path; Path("/tmp/strata-pr1131-prompt.txt").write_text("198 " * 12288)'. The engine command used --pack $DATA/packs/iq3_xxs, the IQ3_XXS native GGUF shard pair, --expert-profile data/expert-profile.bin, --expert-cache auto, --prefill auto, --spec 4, --mtp $DATA/mtp/rt, --max-context 32768, --max-new 1, --kv int8, --no-token-graph, and --tokens-file /tmp/strata-pr1131-prompt.txt.

**Failed:**

- Baseline reproduction on 90767252 failed as described above; no failure was observed on the tested final revision for the supported run.

**Not run:**

- HIP and SYCL builds: hipcc, icpx and dpcpp are unavailable, so this PR is a draft.
- --no-capture: current main rejects this native-pack configuration because session_token has no CPU expert-pool hook; the suggested --no-pool combination was not tested.
- Multi-GPU, two-request serve run, and full CTest suite.

## Limits

The hardware run covers one CUDA GPU and one IQ3_XXS pack. It makes no performance claim and does not establish HIP, SYCL, multi-GPU, or repeated-serve behavior.

No site

Links install, modelos, releases.