#6349 [Engine] Initial prefill batch length changes layer-3 KV and greedy termination with zero checkpoint reuse

open · @ezutfen · 2 commentaires · Sur GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindows

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened
With a cold Qwen3.8-Flash-Next tool-result prompt, changing the initial batched prefill length from 349 to 324 changes the stored int8 KV for the same 324-token prefix and flips a later greedy decision from end-of-turn to `</think>`. No checkpoint was restored in either arm.

### Smallest steps or command that reproduce it

Use the exact prefill IDs below, followed by the held-back ID `271` (one 350-ID request, 349 IDs through the prefill loop). No retokenization. Use greedy decoding (temperature 0, seed 42, 128-token cap). In `engine/src/program/generate.cpp`, for this cold request, set the prefill-loop cut list to `{349}` for A and `{324,343,349}` for B; force `windows_ok` false for these ranges so all prefill segments use the batched path. Keep request IDs, weights, placement, and decode path unchanged; assert `resume=read_from=0` and `cache_n=0`. The original diagnostic used the existing batched prefill call, with no math changes. This is a diagnostic cut override, not a stock CLI reproducer.

```text
[
248045, 8678, 198, 2, 13455, 271, 2523, 599, 2528, 310, 279, 2614, 5568, 25, 271, 27,
15449, 29, 198, 4754, 591, 763, 328, 851, 71877, 487, 328, 4532, 763, 328, 4274, 264,
26388, 27502, 12149, 539, 803, 487, 328, 13390, 763, 5046, 1267, 763, 328, 1640, 487, 328,
12811, 763, 5046, 591, 763, 5046, 1267, 763, 328, 889, 8934, 2069, 328, 6081, 763, 4241,
591, 7664, 328, 34325, 7654, 763, 867, 3307, 198, 510, 15449, 29, 271, 2592, 488, 4992,
310, 1562, 264, 709, 25835, 9559, 303, 279, 2614, 3443, 440, 5486, 19900, 25, 271, 248058,
198, 27, 1628, 28, 8422, 8901, 1224, 29, 198, 27, 15704, 28, 8422, 24109, 62, 16,
29, 198, 927, 62, 16, 198, 510, 15704, 29, 198, 27, 15704, 28, 8422, 24109, 62,
17, 29, 198, 1919, 369, 279, 869, 364, 279, 2018, 5555, 198, 8761, 628, 9111, 198,
34493, 4965, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059, 271, 27, 95328, 29,
198, 92065, 25, 198, 12, 5534, 6526, 26834, 1732, 279, 5024, 3443, 25, 449, 8906, 361,
1628, 28, 1076, 1419, 1628, 29, 2424, 1902, 381, 23283, 2785, 220, 248058, 248059, 11535, 9212,
198, 12, 12296, 4868, 26834, 381, 5024, 198, 12, 1394, 1189, 3300, 9801, 31626, 364, 678,
709, 1562, 303, 5629, 3992, 54588, 279, 709, 1562, 11, 694, 4045, 1238, 198, 12, 1368,
1017, 369, 874, 709, 1562, 2420, 11, 4087, 279, 3296, 1040, 4472, 440, 678, 1428, 6337,
321, 635, 524, 3184, 279, 1156, 883, 709, 6526, 198, 510, 95328, 29, 271, 17, 15464,
2680, 10020, 6481, 13, 10789, 279, 1156, 1622, 22898, 13, 248046, 198, 248045, 846, 198, 6994,
1301, 71877, 440, 803, 869, 13038, 3778, 13, 4486, 1141, 1067, 11, 9559, 6681, 440, 1141,
869, 13, 248046, 198, 248045, 74455, 198, 248068, 271, 248069, 271, 248058, 198, 27, 1628, 86779,
71877, 29, 198, 27, 15704, 37943, 29, 198, 26157, 3778, 198, 510, 15704, 29, 198, 510,
1628, 29, 198, 248059, 248046, 198, 248045, 846, 198, 248066, 198, 33, 62639, 7049, 5228, 40112,
12, 22, 59799, 198, 248067, 248046, 198, 248045, 74455, 198, 248068, 271, 248069
]
```

### Observed on the diagnostic build
- A `[0,349)`: expected `BENCH-FIXTURE-7KR` plus newline, 2/2. B `[0,324)`, `[324,343)`, `[343,349)`, all batched: 0/2 exact; the answer was followed by a literal `</think>` and a duplicate answer. Separate controls using verifier windows for the suffix also failed, but those windows are not required.
- Layer 3 (zero-based, first QSA layer on GPU 0): hash the valid int8 KV rows `[0,324)` in A after the full batch and in B immediately after its first batch, before the suffix. K/V/scale byte digests differ on that common extent. Example FNV-1a K: `2db630890be0e9d1` vs `502a16bb6533fe01`; V: `53cc35ce195923a9` vs `55eec8e0facd7144`. This is not a comparison of recurrent state at different positions.
- Raw matched-history logits in one run (end-of-turn/EOS `248046`, close-think `248069`): at verifier position 349, A `(6.570006, 6.379184)`, B `(6.164159, 6.869490)`; both selected ID `33`. At position 358, A `(19.665434, 17.208570)` selected `248046`; B `(19.379753, 19.615484)` selected `248069`.

**Interpretation:** The initial batch shape is sufficient to make the common-prefix KV differ before the suffix; the all-batched split with zero reuse is sufficient to trigger the output failure. This does not establish a faulty kernel, a race, or that the measured KV difference alone causes the later logit flip. Expected: changing segmentation should not turn the same greedy tool follow-up into a malformed duplicate. Related: #4059 (repeat-request logit drift), #5626 (timing-driven CPU prefill share, disabled in this test); neither establishes the cause here.

### Reproduction status

Reproduced locally

### Strata version

CUDA engine 0.1.62 source base `18526b879231dd5d9c4069c4560c132cc2d2d0b1`, with diagnostic-only cut/trace hooks.

### Graphics card, memory and system

Debian 13 x86-64, Xeon E5-2650 v4, 2 × RTX 3060 12 GB (SM 8.6), NVIDIA driver 610.57.04, approximately 252 GiB system RAM.

### Model, size and settings

Qwen3.8-Flash-Next Unsloth UD-Q5_K_XL native GGUF/pack. One request slot, `--expert-cache auto --prefill auto --layer-split auto --max-context 32768 --kv int8 --ple-io ram --spec 4 --adapt-swaps 0 --pcie-frac 0`, MTP off (no `--mtp`); `STRATA_PREFILL_CPU_SHARE=0 STRATA_MMQ_KQ=1 STRATA_ROUTE_TAIL_SKIP=0 STRATA_ROUTE_PRIOR=0 STRATA_KVA=0`.

Sur le site

Liens install, modèles, releases.