#6441 [Engine] Reduce resident Metal QSA prefill scratch
open · draft · @Maryarty · 0 评论 · 去 GitHub 看
Setup & installAMD / HIPModels & quants
说明
## Problem and result Resident Metal prefill reserves QSA scratch for the full prompt chunk. Two allocations also belong to paths that can be inactive in a given workspace. On my M1 Max, bounding attention partials and allocating just the live buffers reduced owned prefill scratch from **1337.55 MB to 966.28 MB** at a 512-token chunk. ## Approach - `STRATA_METAL_PROMPT_QSA_LIVE=1` allocates the FP16 query buffer `qcurh` for the matrix-unit attention path and the activation pack `XQ` for the legacy merge. - `STRATA_METAL_PROMPT_QSA_SLICE=1` uses 128-row attention/merge slices on the M1 non-matrix-unit path. They reuse the partials buffer while keeping KV positions global. Query preparation, index selection and the output projection retain their existing main chunk. With both flags, the legacy activation pack fits one slice. Both options default to off. The growth test verifies the resulting ownership, the partials bound, and the legacy pack size. Related scratch backing reuse is in #6432. ## Validation M1 Max, 32 GPU cores, 64 GiB, macOS 26.6.2 (25G83), base `3142b66`, head `dae3245`. Release Metal build passed. `ctest --test-dir engine/build-metal -R '^(prompt_workspace_growth|runtime_hello)$' --output-on-failure` and `engine/build-metal/qsa_prompt_parity` passed on the PR branch. The full-model validation build (`0830ead`) adds two resident weight releases from #6440 and a scratch size log. Both weight releases were enabled in every arm to keep the model within the memory guard. Every run used Qwen3.8-Flash-Next GSQ-RCO Q2_0 with PLE, 2,048 fixed input IDs, context limit 2,304, MTP width 2, and 128 greedy output tokens. | QSA setup | Owned prefill scratch | | --- | ---: | | Both switches off | 1337.55 MB | | Live allocations only | 1280.14 MB | | 128-row slices only | 1023.69 MB | | Both switches, direct FP16 merge | 966.28 MB | | Both switches, legacy merge | 979.06 MB | **Passed.** All five arms completed with identical first-logit hashes, all 128 output IDs and MTP counts. Zero new system swapouts. The unmodified allocation arm compiled Metal libraries cold, so this comparison reports memory and parity. **Not run.** Full-model A/B on an exact PR binary with the two weight releases absent, and on other Metal devices or real model formats. This stays in draft for wider coverage. ### Three growing resident turns from 16K On the same validation build with both weight releases common, the 128-token chunk with QSA switches off took 95.99 s to the first token on the initial 16,384-ID request. The 256-token chunk with both QSA switches on took 71.06 s. Both configurations completed three growing turns through 17,280 input IDs. Every corresponding turn matched the first-logit hash, all 128 output IDs and MTP counts. Both recorded zero new system swapouts, with minimum free memory of 17% and 16% respectively. This pair changes the chunk and both switches together. The fixed-512 matrix above isolates scratch savings from each switch.
本站相关内容
相关页面的快捷入口。