#6415 [Engine] Skip older draft rows in resident Metal prefill

open · draft · @ain-soph · 0 comentários · No GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quants

Descrição

## Problem and result

The resident Metal MTP drafter prepares every prompt row, even rows its attention window cannot reach from a saved checkpoint or the final prompt. With `STRATA_METAL_MTP_SKIP=1`, a single-slot resident server can omit the old draft K/V rows. The main model still reads the whole prompt. The 106,668-token fixture improved from about 2,326 to 2,475 tok/s on one M5 Ultra.

## Approach

Keep the trailing draft window before the earliest relevant saved checkpoint or the end of the prompt. Keep the dispatch grouping of retained rows. If a request is cancelled after new prompt tokens have been read, the opt-in path can reuse a saved checkpoint or read the prefix again. It cannot continue from an arbitrary partially read draft state. The flag is off by default. The multi-slot, streaming and one-shot paths do not skip draft rows.

## Validation

**Passed.** Metal builds of upstream `3142b66b` and the candidate `07cb4cac` using `cmake -S engine/metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_GEOMETRY=flash_next -DMETAL_CPP_DIR=<metal-cpp>` and `cmake --build build-metal --target strata`. M5 Ultra, macOS 27.2.1, 96 GB, resident AtomicChat AD-4.27, INT8 KV, Q2_0 MTP, context 131,072, one slot. Every arm used `--prefill 4096`, `STRATA_METAL_PROMPT_CHUNK=1024`, `--spec 3 --spec-min-p 0.5`, `--conversation-cache-mib 1024 --prompt-cache-every 16384`. The table shows the complete run order with a fresh engine and a warm-up in each arm.

| Configuration | 31,323-token prefill seconds | 106,668-token prefill tok/s | Long fixture decode tok/s | Peak wired increment MiB | Peak RSS GiB |
| --- | ---: | ---: | ---: | ---: | ---: |
| main | 13.130 | 2,325.5 | 120.3 | 61,023 | 66.30 |
| PR off | 13.138 | 2,321.4 | 120.3 | 60,945 | 66.30 |
| PR on | 12.320 | 2,475.3 | 120.3 | 61,548 | 66.42 |
| main repeat | 13.096 | 2,326.2 | 119.9 | 61,018 | 66.30 |

All four arms restored the same 16,384-token periodic checkpoint and matched greedy answers byte for byte for the resumed request and both cold long inputs. The opt-in cancellation test closed a streaming request after observing 18,432 prompt tokens. The retry reused 16,384 tokens and returned its expected marker.

**Failed.** None in the final four-arm validation.

**Still to validate.** CUDA/HIP/SYCL compilation for the shared call signature, a second Mac, additional model packs and explicit pinned/checkpoint variants. This remains a draft.

No site

Links install, modelos, releases.