#6136 [Engine] 0.1.60: prompt + appended tail re-reads a whole equal chunk (append 6.4x slower than 0.1.54, first read 7.5% faster)
open · @gaoyonghui123 · 2 评论 · 去 GitHub 看
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
说明
## Environment
- Engine: **0.1.60** (`strata-engine-0.1.60-windows-x86_64-cuda-cuda13`, BUILD.json commit `09b603886925`)
- Previous engine for comparison: **0.1.54** (same model, same config, same machine, back to back)
- GPU: RTX 4080 16 GB (sm_89) / Ryzen 7 7800X3D / 64 GB DDR5 / Windows 11 / driver 617.42
- Model: Qwen3.8-Flash-Next GSQ-RCO **IQ3_XXS**, native pack + PLE, MTP `--spec 6`
- Launch: `--pack … --native … --ple-gguf … --expert-profile … --expert-cache auto --prefill auto:32768 --spec 6 --mtp … --max-context 262144 --kv int8 --mmap-experts --spec-min-p 0.7 --pool-workers 5 --host-core sibling --resident-budget-gib 40 --pcie-frac 0.5 --vram-reserve-mib 768 --kv-resident 20480 --conversation-cache-mib 8192 --conversation-cache-slots 4 --conversation-cache-min-free-mib 512 --conversation-cache-min-tokens 20000 --conversation-cache-park-gain 4000`
## What happens
A long prompt followed by a **short appended tail** (the normal "second turn of a long conversation") re-reads a whole equal chunk instead of a small remainder.
66,968-token prompt, then the same prompt with ~9 tokens appended. Measured with our own harness (`serve/server.py --engine strata`, `/v1/chat/completions`), each arm a fresh engine start, 2 runs per arm:
| | 0.1.54 | 0.1.60 | change |
|---|---:|---:|---:|
| first read, 66,968 tok | 3,554–3,592 tok/s | **3,852–3,862 tok/s** | **+7.5%** |
| same prompt resent | 715–748 ms | **41–44 ms** | **17× faster** |
| **prefix + 9-token tail** | **810–853 ms** | **5,276–5,296 ms** | **6.4× slower** |
| weighted decode (3 scenes 93/5/2) | 98.8–109.9 tok/s | 110.1–113.7 tok/s | +7.0% |
So the same cache is 17× faster for an *identical* prompt and 6.4× slower for *prefix + tail*. Every turn of a long conversation pays ~4.4 s more.
## Engine log evidence
Same run, three requests:
```
first : prompt 66968 tokens = 0 reused + 66968 read in 17318 ms, 4 checkpoints
resend : prompt 66968 tokens = 66963 reused + 5 read in 37 ms, 4 checkpoints
append : prompt 66977 tokens = 50223 reused + 16754 read in 5237 ms, 4 checkpoints
```
The append re-reads **16,754 ≈ 66,977 / 4** — exactly one equal chunk. 0.1.54 for the same case (its own log line, same harness) re-reads ~1,000 tokens.
## Root cause (I read v0.2.14 source, `git clone --depth 1 --branch v0.2.14`)
`include/strata/prefill/equal_cut_rule.hpp:103-112`, `equal_chunk()`:
```cpp
const int64_t n = chunk_count(tokens, max_chunk);
const int64_t per = tokens / n + (tokens % n != 0); // <- per depends on `tokens`
const int64_t last = tokens - (n - 1) * per;
```
`per = ceil(tokens / n)` changes whenever `tokens` changes, so **every chunk boundary moves** when a few tokens are appended.
Checkpoints are taken only at chunk boundaries — `src/program/generate.cpp:13693`:
`const int64_t done = p0 + T;` (T = the whole chunk) and `periodic_checkpoint = … done >= pp_next_check …`; the other hook, `src/prefill/prefill.cpp:7325`, is `on_chunk(m.R, T, p0, err)` — also whole-chunk only.
So the last checkpoint sits at `(n-1) * per`, which shifts by the same amount the tail grows, and everything after it is re-read: an entire `L/n` chunk instead of a remainder.
0.1.54 used `Cut::today` ("full chunks and a remainder"), whose chunk size is the configured chunk and does **not** move, so only the remainder is re-read.
## What I tried, and what it did
All on 66,968-token prompt + 9-token tail, engine 0.1.60, same harness:
| Change | first read | append | re-read tokens |
|---|---:|---:|---:|
| (baseline) | 20,473 ms | 5,960 ms | 16,754 |
| `--prompt-cache-every 8192` | 20,320 | 6,059 | – |
| `--prompt-cache-spread` | 20,124 | 5,938 | – |
| `--prompt-cache-tail` | 20,608 | 6,065 | 16,754 |
| `--prompt-cache 12` | 20,615 | 6,037 | 16,754 (still 4 checkpoints) |
| `--prefill 8192` | 26,134 | 9,251 | 22,331 |
| `--prefill 6144` | 29,281 | 6,262 | 12,185 |
| **`STRATA_PREFILL_EQUAL=0`** | 20,810 | 2,629 | 1,697 |
| **`EQUAL=0` + `STRATA_PREFILL_CPU_SHARE=0`** | 21,236 | **1,490** | **161** |
Notes:
- The number of checkpoints is fixed by the chunk count, not by `--prompt-cache` (setting it to 12 still logs `4 checkpoints`).
- `--prompt-cache-tail` is documented as "at an **existing chunk boundary**", so it cannot help here either.
- Making the chunk smaller makes the boundary drift worse (`--prefill 6144` lost two chunks) and the first read 44% slower.
## Where it still falls short
With `STRATA_PREFILL_EQUAL=0` + `STRATA_PREFILL_CPU_SHARE=0`, the standard-harness numbers are:
| | 0.1.54 | 0.1.60 + both switches |
|---|---:|---:|
| first read | 3,554–3,592 tok/s | **3,848–3,899 tok/s (+9.1%)** |
| append | **810–853 ms** | **1,481–1,497 ms (still +79%)** |
The remaining ~650 ms is a **fixed cost in the small-tail resume path**: the log shows `66816 reused + 161 read in 1539 ms` — 161 tokens taking 1.5 s (~9 ms/token) while the first read runs at 0.25 ms/token. So closing the gap needs work inside the resume/prompt path, not a config change.
## Why I am not filing a patch
Checkpoints can only exist at chunk boundaries because the K/V is computed layer by layer; mid-chunk the state is incomplete. So "decouple the checkpoint from the chunk boundary" is not a small change, and the remaining fixed cost is in the kernel. I would rather report it than carry a fork.
## Ask
Two things would make 0.1.60 usable for long conversations:
1. Let the last chunk be a remainder (or at least keep chunk boundaries stable when `tokens` changes by a few), so an appended tail re-reads a remainder instead of `L/n`.
2. Look at the ~1.5 s fixed cost of resuming with a small (161-token) tail — that is what keeps the switch-assisted append at 1.5 s instead of 0.85 s.
The 0.1.60 first-read gain (+7.5% here) and decode are real and worth having; it is only the append path that keeps us on 0.1.54.
---
Harness: our own Python script driving `serve/server.py` over HTTP (`/v1/chat/completions`, greedy, temperature 0), a fresh engine start per arm, 2 arms per configuration, machine otherwise idle (53.9 GiB of 63.2 GiB free). Raw logs and result JSON available if useful.
本站相关内容
相关页面的快捷入口。