#6339 [Engine] 0.1.62: --lookup-max-t past 8 with --draft dflash2 ends the request (dflash2: draft rows out of range); a small fix, +11..13 % on file rewrites
open · @router0mail · 4 comentários · No GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsWindowsLinux
Descrição
### Setup
Linux 7.0, Ryzen 9 9950X, 128 GB RAM. RTX 3090 Ti 24 GB (sm_86, CUDA 13.2, on a PCIe x8 link); its clock is capped at 1770 MHz and its tensor cores are unlocked with gpu-unlock v2.0.0. Strata v0.2.16 built from source (engine 0.1.62). Qwen3.8-Flash-Next GSQ-RCO IQ3_S (native GGUF and pack), the experts that do not fit stay in RAM (`--ple-io ram --expert-cache auto`: 8601 slots), `--kv int8 --kv-resident 32768 --max-context 262144 --vram-elastic`, one request slot. The server starts DFlash2 as the only drafter: `--draft dflash2 --draft-model ... --draft-fallback none --spec 6 --draft-conf 0.7`.
### What happens
`core/lookup_default.hpp` keeps the old windows when the drafter is DFlash2, so the long prompt-lookup windows are reached only with an explicit `--lookup-max-t`. Given `--lookup-max-t 12` (or 16), the engine accepts it and sets everything up:
```
strata verify: window up to 12 tokens, 111.0 MiB of device buffers
strata verify: capturing the 12-token window (615 MiB of VRAM free)
strata serve: --lookup-max-t 12: MTP windows up to 6 tokens, window sizes 1,2,3,4,5,6,7,8,12
```
The first lookup window that commits more than 8 rows then ends the request (a whole-file rewrite stopped after 38 tokens), and the server reports
```
[strata] the engine reported an error: dflash2: draft rows out of range
```
The verify loop hands the drafter every accepted row, and `Dflash2Drafter::draft_launch` takes at most `kRows` (8): `if (n_new < 1 || n_new > kRows)`. The MTP drafter has `draft_long` for such a window; DFlash2 has nothing in its place. `main` at 58ee322772 still has the same check.
### A fix
`Dflash2Drafter::draft_rows`: for more than `kRows` committed rows, the rows before the last go into the ring as context rows (the existing `context()`), the last row is copied to row 0 of the taps (the buffer the round's graph was captured on) and the usual one-row round runs; the copy and the graph are on the drafter's own stream, so they are ordered. A window of up to 8 rows calls `draft()` exactly as before. The four `df2.draft` calls of the serve and generate loops go through it. Row 0 of the verifier's tap buffer is overwritten after the drafter has read it; nothing reads it afterwards on one card (a layer split and `--draft auto` keep windows at 8 rows, so they do not get there).
<details><summary>diff against v0.2.16 (92 lines)</summary>
```diff
diff --git a/engine/include/strata/core/dflash2.hpp b/engine/include/strata/core/dflash2.hpp
index 5fbf17579..cd5878ecb 100644
--- a/engine/include/strata/core/dflash2.hpp
+++ b/engine/include/strata/core/dflash2.hpp
@@ -136,6 +136,10 @@ public:
/// then the block for `anchor` (the last committed token, at pos0 + n_new). `drafts` / `probs` get kDrafts entries (the
/// selector's pick and its probability). The taps pointer must be the same device buffer on every call (graphs are captured on it).
bool draft(const float* taps_dev, int n_new, int64_t pos0, int32_t anchor, int32_t* drafts, float* probs, std::string& err);
+ /// draft() for a window of more than kRows committed rows (--lookup-max-t past kVerifyNarrowT: a prompt-lookup window of up to kVerifyLongT rows):
+ /// rows 0 .. n_new - 2 become context rows, then the round runs for the last row alone, which is copied to row 0 of `taps_dev` first (row 0 is
+ /// overwritten). n_new <= kRows is draft() unchanged. Afterwards the ring holds every committed row, as after a shorter window.
+ bool draft_rows(float* taps_dev, int n_new, int64_t pos0, int32_t anchor, int32_t* drafts, float* probs, std::string& err);
/// draft() in two halves: draft_launch queues the round's graph and returns, draft_finish waits and reads the picks. Several drafters (batch slots, each on its
/// own stream) can launch first and finish after, so their graphs run at the same time.
diff --git a/engine/src/core/dflash2.cpp b/engine/src/core/dflash2.cpp
index eb96c0437..50b7a9ce6 100644
--- a/engine/src/core/dflash2.cpp
+++ b/engine/src/core/dflash2.cpp
@@ -1016,6 +1016,20 @@ bool Dflash2Drafter::draft(const float* taps, int n_new, int64_t pos0, int32_t a
return draft_launch(taps, n_new, pos0, anchor, err) && draft_finish(drafts, probs, err);
}
+bool Dflash2Drafter::draft_rows(float* taps, int n_new, int64_t pos0, int32_t anchor, int32_t* drafts, float* probs, std::string& err) {
+ if (n_new <= kRows) return draft(taps, n_new, pos0, anchor, drafts, probs, err);
+ // the rows before the last go into the ring as context rows (chunks of kRows); the round then takes the last row alone, moved to row 0
+ // (its graph reads the buffer it was captured on). The ring ends up as one round over all n_new rows would leave it.
+ if (!context(taps, n_new - 1, pos0, err)) return false;
+ const OnDevice on_device(device_);
+ const size_t row = taps_.size() * (size_t) N_;
+ if (cudaMemcpyAsync(taps, taps + (size_t) (n_new - 1) * row, row * sizeof(float), cudaMemcpyDeviceToDevice, cs_) != cudaSuccess) {
+ err = std::string("dflash2: ") + cudaGetErrorString(cudaGetLastError());
+ return false;
+ }
+ return draft(taps, 1, pos0 + n_new - 1, anchor, drafts, probs, err);
+}
+
bool Dflash2Drafter::draft_launch(const float* taps, int n_new, int64_t pos0, int32_t anchor, std::string& err) {
if (!bound_) { err = "dflash2: not bound"; return false; }
if (!context_wait(err)) return false;
diff --git a/engine/src/program/generate.cpp b/engine/src/program/generate.cpp
index 809c31e3d..1dffdaa58 100644
--- a/engine/src/program/generate.cpp
+++ b/engine/src/program/generate.cpp
@@ -15675,8 +15675,9 @@ int main(int argc, char** argv) {
}
}
if (o.lookup_max_t > 0)
- std::fprintf(stderr, "strata serve: --lookup-max-t %d: MTP windows up to %d tokens, window sizes %s\n",
- o.lookup_max_t, S_mtp, size_list(policy.sizes(), S).c_str());
+ std::fprintf(stderr, "strata serve: --lookup-max-t %d: %s windows up to %d tokens, window sizes %s%s\n",
+ o.lookup_max_t, use_df2 ? "DFlash2" : "MTP", S_mtp, size_list(policy.sizes(), S).c_str(),
+ use_df2 && S > strata::kernels::kVerifyNarrowT ? " (a lookup window past 8 rows: the drafter takes its rows as context first, the last row as its round)" : "");
// --lookup-max-t: a lookup window's graph is captured before its round; a size whose graph finds no VRAM is
// dropped for good (the policy never picks it again) and the round takes the next smaller allowed size.
// -1: a capture failed for another reason (err says why).
@@ -21437,7 +21438,7 @@ int main(int argc, char** argv) {
int32_t dd[strata::core::Dflash2Drafter::kDrafts];
float pp[strata::core::Dflash2Drafter::kDrafts];
const float* tp = df2_win_taps(a_ + 1, err);
- if (tp == nullptr || !df2.draft(tp, a_ + 1, p, outv[(size_t) a_], dd, pp, err)) return false;
+ if (tp == nullptr || !df2.draft_rows(const_cast<float*>(tp), a_ + 1, p, outv[(size_t) a_], dd, pp, err)) return false;
for (size_t i = 0; i < drafts.size() && i < (size_t) strata::core::Dflash2Drafter::kDrafts; ++i) { drafts[i] = dd[i]; dprob[i] = pp[i]; }
if (ss.ple.ready() && ss.ple.table != nullptr) { // the PLE rows of the next window's tokens, read ahead (as the MTP's draft does)
int n_pf = 0;
@@ -21481,7 +21482,7 @@ int main(int argc, char** argv) {
} else {
int32_t dd[strata::core::Dflash2Drafter::kDrafts];
float pp[strata::core::Dflash2Drafter::kDrafts];
- if (!df2.draft(ver.tap_dev(), dfr_n, dfr_pos, dfr_anchor, dd, pp, err)) { std::printf("ERR %s\n", err.c_str()); return 1; }
+ if (!df2.draft_rows(const_cast<float*>(ver.tap_dev()), dfr_n, dfr_pos, dfr_anchor, dd, pp, err)) { std::printf("ERR %s\n", err.c_str()); return 1; }
for (size_t i = 0; i < drafts.size() && i < (size_t) strata::core::Dflash2Drafter::kDrafts; ++i) { drafts[i] = dd[i]; dprob[i] = pp[i]; }
}
}
@@ -23684,7 +23685,7 @@ int main(int argc, char** argv) {
if (wait_commit_first && !ver.wait_commit(err)) return false;
int32_t dd[strata::core::Dflash2Drafter::kDrafts];
float pp[strata::core::Dflash2Drafter::kDrafts];
- if (!df2.draft(ver.tap_dev(), a_ + 1, p, outv[(size_t) a_], dd, pp, err)) return false;
+ if (!df2.draft_rows(const_cast<float*>(ver.tap_dev()), a_ + 1, p, outv[(size_t) a_], dd, pp, err)) return false;
for (size_t i = 0; i < drafts.size() && i < (size_t) strata::core::Dflash2Drafter::kDrafts; ++i) { drafts[i] = dd[i]; dprob[i] = pp[i]; }
// what the MTP does while it drafts: the PLE rows of the tokens the next window will hold are read ahead
// (the window's gather would otherwise wait for them: 1.8 ms a round on strix)
@@ -23731,7 +23732,7 @@ int main(int argc, char** argv) {
} else {
int32_t dd[strata::core::Dflash2Drafter::kDrafts];
float pp[strata::core::Dflash2Drafter::kDrafts];
- if (!df2.draft(ver.tap_dev(), dfr_n, dfr_pos, dfr_anchor, dd, pp, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; }
+ if (!df2.draft_rows(const_cast<float*>(ver.tap_dev()), dfr_n, dfr_pos, dfr_anchor, dd, pp, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; }
for (size_t i = 0; i < drafts.size() && i < (size_t) strata::core::Dflash2Drafter::kDrafts; ++i) { drafts[i] = dd[i]; dprob[i] = pp[i]; }
}
}
```
</details>
### Numbers
Same build with and without `--lookup-max-t 16 --lookup-sizes 7,8,12,16`, a fresh engine an arm, two runs an arm, interleaved, greedy, thinking off.
A whole-file rewrite with one rename (2,401-token prompt, 2,354 new tokens, three in a row):
| | old windows | `--lookup-max-t 16` |
|---|---|---|
| first rewrite on a fresh engine | 185 / 186 tok/s | 188 / 188 |
| second and third | 185-192 | 209-213 |
| tokens a window | 6.3-6.8 | 9.7-12.6 |
| ms a window, of it missed experts on the CPU pool | 34-37, 12-13 | 49-61, 19-26 |
The answer is the same text in every run of both arms (one hash), and it is the source with the rename applied.
Where the answer is new text nothing moves: 24 coding tasks (bugs / tests / refactor over 2-9K-token files, 700 new tokens each) 141 / 145 tok/s as a set against 143 / 146 (with other changes in that arm; single tasks differ by more than that between two runs of the same arm); 256-token essays 100.3 / 101.0 against 100.5 / 99.3 tok/s; 1024-token essays 112.5 / 113.1 against 117.2 / 117.2.
It costs about 100 MiB of free VRAM (the wider window buffers and the 12- and 16-token graphs: "761 MiB of VRAM free with everything loaded" becomes 713, then two more graphs); the expert cache keeps its 8601 slots. The window warm-up reads `8:41 12:48 16:62` ms per width.
Also tried: the wide build (`-DSTRATA_WINDOW_MAX_T=16`) was slower on the same rewrite, 187 / 198 tok/s against 205 / 207 (its 16-row window warms up at 79 ms against 61-62 ms for the two groups of 8), and `--lookup-chain 4` after DFlash2's drafts did not help (it fired in 18 of 413 rewrite windows and 20 of 239 coding windows).
With long windows the CPU pool's share of a window grows from about a third to 40 %, so on this card the next step for copy-heavy answers would be in how the missed experts of a wide window are divided between the pool and the link.
A side note for #5449 (the 12-token window faulting on an RTX 3090 with an IQ3_S native pack on 0.1.55-0.1.58): here, on 0.1.62 with the same class of No site
Links install, modelos, releases.