#6432 [Engine] Reuse Metal resident prefill scratch buffers

open · @Maryarty · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quants

Description

## Problem and result

Resident Metal prefill allocates QSA partials (`part`), gathered expert input (`XS`) and expert output (`YE`) separately. Those buffers are live at different points in the serial prompt chain. With `STRATA_METAL_PROMPT_REUSE=1`, they share backing storage sized for the largest view. On my M1 Max, this saved **165 MB of peak process footprint** at a 256-token chunk.

## Approach

The QSA merge consumes `part` before the expert gather writes `XS`. Gate/up consumes `XS` before down writes `YE`. Only the backing allocation is registered for release. The existing Metal workspace test now checks that these three views alias only with reuse enabled, that same-size requests keep their buffers, and that old allocations are released when the workspace grows. The option defaults to off.

Related workspace lifetime work: #3527.

## Validation

M1 Max with 32 GPU cores and 64 GiB, macOS 26.6.2 (25G83), Metal engine 0.1.63. Tested upstream `3142b66` and local `dd51957`. The submitted engine code matches the tested patch. The follow-up commit changes only the test.

**Passed.** Release Metal builds for both revisions. On the patched build, `ctest --test-dir engine/build-metal -R 'prompt_workspace_growth|runtime_hello' --output-on-failure` passed both tests.

I used the same Qwen3.8-Flash-Next GSQ-RCO Q2_0 pack with PLE for each run. Each arm starts a fresh `strata --serve` process with `--prefill auto --spec 2 --suffix-draft 0 --max-context 2304`, `STRATA_METAL_STREAM=0` and `STRATA_METAL_PROMPT_CHUNK=256`, then sends the same 2048 code-review input IDs and generates 128 greedy tokens. Peak footprint was sampled using `proc_pid_rusage`.

| Build | Peak process footprint | TTFT | Decode |
| --- | ---: | ---: | ---: |
| Upstream | 6841.4 MB | 8.58 s | 47.79 tok/s |
| Patch, reuse off | 6841.2 MB | 8.54 s | 47.94 tok/s |
| Patch, reuse on | 6676.3 MB | 8.56 s | 48.11 tok/s |

All three completed with identical 128 output IDs, MTP acceptance of 54/73, and zero new system swapouts.

**Failed.** An earlier matrix at context limit 4608 and chunk 512 was incomplete. The memory guard stopped the patched/reuse-off arm after 5304 new system swapouts. The stock and reuse-on arms completed with identical output IDs. I do not attribute those swapouts to a particular allocation.

**Not run.** Real-model runs on other quantizations and GPU generations. CUDA, HIP and SYCL are outside the Metal-only change. The completed comparison has one run per arm and supports the footprint and output comparison. I am keeping the PR in draft for wider parity coverage.



### Workspace growth follow-up

The lifetime test now disables test-process keep-alive so its deallocation assertions run without background command buffers in flight. I observed an intermittent retention at final free with keep-alive on. With the test-only isolation, `prompt_workspace_growth` and `runtime_hello` each passed three consecutive runs on the M1 Max.

The repeated full-model growth series hit the memory guard in stock and both patched modes. Those runs added zero swapouts and do not give a complete session comparison. The full-model footprint and output evidence remains the three-arm one-request comparison above.


### Completed full-model workspace growth with a shared memory baseline

I repeated the comparison with both resident weight releases from #6440 enabled in every arm. The control is `0d91f17` from upstream `3142b66`. The comparison build `a49fa7b` contains the #6432 engine changes and the disabled M8 flag. Each fresh process used the same M1 Max, Q2_0 and PLE, MTP width 2, context 2304 and chunk 256. The workspace grew from 120 to 256 rows in all three arms, on inputs of 127, 639 and 1279 IDs.

The control and both patched modes finished all three requests. Every corresponding request matched the first-logit hash, all 128 output IDs and the MTP counts. Zero new swapouts. The minimum system free memory was 17% for the control and 18% for both patched modes. At the end of the three requests the reuse-on footprint was 184, 204 and 204 MB below the reuse-off arm. Sampled process peaks were similar across the whole run. The earlier three-arm result above covers the exact upstream/PR binaries without the common releases.

Sur le site

Liens install, modèles, releases.