#6052 [Engine] Equal prompt chunks skip periodic conversation checkpoints (--prompt-cache-every)
closed · @Tsury · 3 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUNVIDIA / CUDALinux
Description
With the equal prompt cut that v0.2.14 turns on for a build without MMQ on one GPU, the periodic conversation checkpoints (`--prompt-cache-every`) land far less often than the spacing asks. A follow-up turn on a long conversation then reuses about half of the prompt instead of all but the last chunk. Environment: Strata v0.2.14 `619a1b43` (engine 0.1.60), build without MMQ (own kernels, `STRATA_PF_FUSED=1` default), Linux, CUDA 13.4, one RTX 5090 Laptop GPU (24 GB), `--prompt-cache 6 --prompt-cache-every 16384 --prefill auto:32768`. Same model, files and settings on v0.2.13 for comparison. ## What happens A 61,644-token prompt, then the same conversation with about 4K new tokens appended: | | prompt chunk | checkpoints on the first prompt | follow-up turn | |---|---|---|---| | v0.2.13 | auto 16384 | 4 | 49152 reused + 15603 read, first output 8.1 s | | v0.2.14, `auto:32768` | auto 32768 (2 equal chunks of ~30.8K) | 2 | 30820 reused + 33935 read, first output 13.0 s | | v0.2.14, `auto:16384` | 16384 (4 equal chunks of 15,412) | 2 | 30822 reused + 33936 read, first output 12.3 s | | v0.2.14, `auto:16384` + `--prompt-cache-every 8192` | 16384 (4 equal chunks) | 4 | 46233 reused + 18526 read, first output 7.6 s | Engine lines (v0.2.14, `auto:16384`, spacing 16384): ```text strata serve: prompt chunk auto: 16384 tokens, a 96-slot ring strata serve: prompt 61647 tokens = 0 reused + 61647 read in 20356 ms (3028.4 tok/s), ... 2 checkpoints strata serve: prompt 64758 tokens = 30822 reused + 33936 read in 12211 ms (2779.2 tok/s), ... 3 checkpoints ``` ## Cause `src/program/generate.cpp` takes a periodic checkpoint only at a chunk end, and the next one is due `prompt_cache_every` tokens after the chunk end that took it (v0.2.14, lines 13703 and 13735): ```cpp const bool periodic_checkpoint = o.prompt_cache_every > 0 && done >= pp_next_check && !sp.layer_major_mid(); ... if (periodic_checkpoint) pp_next_check = done + o.prompt_cache_every; ``` Before the equal cut, chunk ends fell on multiples of the chunk size, which matched the spacing. Equal chunks are slightly shorter than the cap, so with spacing = cap every other chunk end is just under `pp_next_check` (15412 < 16384, then 46235 < 30823 + 16384) and is skipped. With the default `auto:32768` cap, the two ~30.8K chunks leave a single mid-prompt checkpoint. ## Suggested fix Schedule the next checkpoint on the spacing grid instead of relative to the chunk end that took the last one, for example: ```cpp if (periodic_checkpoint) pp_next_check = (done / o.prompt_cache_every + 1) * o.prompt_cache_every; ``` With 15,412-token chunks and spacing 16384 this takes checkpoints at 30823 and 46235 as well as the end. The multi-GPU `part_next` schedule uses the same `done + prompt_cache_every` form. ## Workaround A spacing of at most half the chunk cap takes a checkpoint at every chunk end, because an equal chunk is never shorter than half the cap: `--prefill auto:16384 --prompt-cache-every 8192` restores v0.2.13's follow-up reuse (table above). On this machine the 16K cap is also faster than `auto:32768` (64K cold prompt 3050 vs 2710 tok/s), and `--prompt-cache 6` still caps a 144K prompt at 6 checkpoints.
Sur le site
Liens install, modèles, releases.