#6249 [Engine] Opt-in STRATA_PREFILL_EQUAL_TAIL: keep a short last prompt chunk instead of a second streamed pass

open · @woodsonl · 0 Kommentare · Auf GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDADocumentationWindowsLinux

Beschreibung

## Problem and result

On one GPU in a build without MMQ (the release builds), engine 0.1.60 reads a prompt segment that is longer than the chunk in equal chunks, also when the old cut's last chunk would be short. Each equal chunk is a full streamed pass. With a native pack and the CPU share, a read a few tokens over the chunk costs about 2.9 s more than a read that fits.

This PR adds an opt-in switch, `STRATA_PREFILL_EQUAL_TAIL=N`. With it set, a segment whose old last chunk is under N tokens and under the stream floor keeps one full chunk plus that short last chunk, at the scan's chunk. Unset, nothing changes.

Measured on an RTX 4080 SUPER 16 GB (RTX 4060 Ti 8 GB as `--expert-cache-device1`, PCIe 4.0 x16 / x4), Ubuntu, CUDA 13.4, the UD-Q4_K_XL pack, control 0.1.60. Both engines are built from the v0.2.14 source (09b6038) with the release cuda13 flags (`-DSTRATA_MMQ=OFF`, sm_89 only): `base` without the patch, `new` and `tail` with it. `tail` sets `STRATA_PREFILL_EQUAL_TAIL=3072` (the CPU share's floor, so every short last chunk is kept). Auto chunk 16,128 in every arm. Serve path, two requests on one conversation, the second adds the tokens below; times are the `serve: prompt ... read in X ms` lines. Three rounds, each arm started fresh per round, arm order alternating, target order rotating; median of 3.

| Part read (tokens) | Over the chunk | base | new (switch unset) | tail (N = 3,072) | tail vs base |
|---|---|---|---|---|---|
| 15,995 | fits | 4,272 | 4,256 | 4,240 | -0.7% |
| 16,095 | fits | 4,293 | 4,296 | 4,265 | -0.7% |
| 16,194 | 66 | 7,193 | 7,199 | 4,801 | -33.3% |
| 16,595 | 467 | 7,222 | 7,223 | 5,483 | -24.1% |
| 17,094 | 966 | 7,245 | 7,258 | 6,061 | -16.3% |
| 17,594 | 1,466 | 7,276 | 7,279 | 6,450 | -11.4% |
| 18,094 | 1,966 | 7,308 | 7,314 | 6,750 | -7.6% |
| 18,595 | 2,467 | 7,345 | 7,350 | 6,917 | -5.8% |
| 19,094 | 2,966 | 7,374 | 7,378 | 7,096 | -3.8% |
| 19,394 | 3,266 (above the floor) | 7,394 | 7,403 | 7,394 | 0.0% |

Every individual run is in the Validation section. Over the seven over-chunk targets below the floor, median wait for expert copies fell from 1,910 to 1,780 ms and K/V staging from 1,302 to 524 ms (`STRATA_PREFILL_TIMING=1`). No request fell back to the equal cut or logged a #2090 loan message.

`STRATA_PREFILL_EQUAL=0` is not a cure: it also lowers the auto chunk, because the fit_out scan counts the tail again. On the same card with the 0.1.60 release engine and the earlier pack (auto chunk 17,152 with the equal cut, 16,384 with `=0`, one round), reads of 16,951 and 17,082 tokens slowed from 4,340 and 4,342 ms to 5,651 and 5,777 ms.

## Related discussion

- #693 and #903: the equal-chunk rule, which kept a short last chunk when it was under the floor.
- #6052: another side effect of equal chunks (periodic checkpoints).
- #5546: prompt-length crossover reporting.

No open issue covers this case. The table suggests that keeping a sub-floor last chunk (#903's rule) would be faster as a default on this card, but a default change needs agreement first, so this PR keeps it opt-in.

## Approach

- `equal_cut_rule.hpp`: `tail_keep()` parses the variable; `keeps_tail()` tests the old cut's last chunk against N and the floor. `equal_chunk()` returns the old chunk when it holds. `declined_equal()` reports the equal chunk this rule turned down, so the existing `keep_equal` fallback (#2090) keeps the equal cut when the old cut's loan does not fit the lendable slots. The new arguments default to 0.
- `generate.cpp`: reads the variable once and passes it on one GPU only, at the serve lend and generate's prompt path.
- The fit_out tail-0 scan is unchanged, so the auto chunk stays. The short tail's buffers are not in the scan's count; the per-request lend sizes them and falls back to the equal cut when they do not fit.
- Docs: a row in `docs/DETAILS.md` with the numbers above, the register entry, and `docs/SETTINGS.md` from `tools/settings_register.py --docs`. The `STRATA_PREFILL_EQUAL` row in DETAILS.md now says the equal cut is the default and `=0` turns it off.

## Validation

Passed:
- CUDA build of the patch on the v0.2.14 source (CUDA 13.4, gcc 15.2, sm_89), and of the same source without it.
- `tests/prefill/equal_cut_rule_test.cpp` on that source and on this branch (Apple clang and gcc): all checks, 281,212 swept cases. New checks sweep 1 to 80,000 tokens at chunks 8,192 / 10,240 / 17,152 and show `equal_chunk` and `declined_equal` give the same result with the switch at 0 as without the argument.
- `tests/prefill/fit_out_rule_test.cpp` and `tools/test_settings_register.py`.
- This PR's CI: CUDA 12.9, CUDA 13.0, HIP ROCm 7.2, SYCL oneAPI 2026.1, Metal and the CPU presets build; the Linux and macOS suites pass.
- A/B above, three rounds. With the switch unset (`new`), every target is within 0.4% of `base` (largest gaps: -16 ms at 15,995 tokens, +13 ms at 17,094), the auto chunk is the same (16,128) and the cut is the same (one timing part per read, same part sizes).

Individual runs, prompt ms, rounds 1 / 2 / 3:

| Part read | base | new | tail |
|---|---|---|---|
| 15,995 | 4,254 / 4,272 / 4,282 | 4,256 / 4,283 / 4,236 | 4,262 / 4,235 / 4,240 |
| 16,095 | 4,293 / 4,301 / 4,282 | 4,302 / 4,296 / 4,286 | 4,253 / 4,265 / 4,278 |
| 16,194 | 7,190 / 7,193 / 7,202 | 7,199 / 7,199 / 7,197 | 4,782 / 4,801 / 4,827 |
| 16,595 | 7,222 / 7,221 / 7,225 | 7,235 / 7,222 / 7,223 | 5,511 / 5,483 / 5,453 |
| 17,094 | 7,245 / 7,245 / 7,258 | 7,258 / 7,259 / 7,250 | 6,087 / 6,026 / 6,061 |
| 17,594 | 7,276 / 7,275 / 7,281 | 7,284 / 7,278 / 7,279 | 6,440 / 6,450 / 6,470 |
| 18,094 | 7,308 / 7,303 / 7,313 | 7,326 / 7,314 / 7,308 | 6,727 / 6,769 / 6,750 |
| 18,595 | 7,345 / 7,342 / 7,348 | 7,351 / 7,350 / 7,345 | 6,917 / 6,954 / 6,916 |
| 19,094 | 7,374 / 7,372 / 7,375 | 7,387 / 7,378 / 7,373 | 7,132 / 7,096 / 7,095 |
| 19,394 | 7,394 / 7,411 / 7,394 | 7,405 / 7,403 / 7,391 | 7,394 / 7,392 / 7,394 |

Failed: the two Windows jobs `Python terminal/serve/` (1 test) and `host typecheck + test + build` (3 tests). This PR changes no host or serve code; the same two jobs fail on the other open pull requests of the same day (`feat/mmvq-il-q8-0`, `fix-nvfp4-verify-ple-formats`, `fix-nvfp4-convert-temp-file`).

Not run:
- A build of this branch's tip with the new `engine/` layout. The measured engines are the v0.2.14 source with the same three-file patch, which applied without fuzz.
- A byte-for-byte comparison of generated text: answers were 8 tokens, and their hashes varied across rounds within every arm, `base` included.
- Other cards, packs, and N below 3,072.

## Limits

- One machine, one pack. The gain shrinks as the short last chunk grows and reaches zero at the floor; on this card it stayed positive up to 2,966 tokens.
- With the switch on, a request can lend more slots than the auto scan planned for (the tail's rows). It stays within the hard limit, or the equal cut is kept. In this run no request needed the fallback.


Mehr auf der Site

Links zu Install, Modellen, Releases.