Pull requests / #2338
#2338 metal: keep IQ3_XXS resident on 64 GB Macs (wrap a model bigger than maxBufferLength as overlapping buffers)
open · @hokar3361 · 0 コメント · GitHub で見る
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quants
本文
## What this changes
On a 64 GB Mac, the IQ3_XXS pack cannot stay resident in v0.1.44: `Context::wrap` makes one no-copy `MTLBuffer` of the whole model file (47.0 GB), which is larger than the device's `maxBufferLength` (41.75 GB on an M4 Max 64 GB), so it fails with `mtl: wrap 47039873024 failed` (with `STRATA_METAL_STREAM=0`), and the automatic choice streams the experts from the SSD.
This PR lets such a file be wrapped and used in place:
- `metal/src/mtl_rt.{h,mm}`: a new `wrap(ptr, bytes, max_span)`. When `bytes > maxBufferLength`, the file is wrapped as several no-copy buffers that overlap by `max_span` (the largest range ever bound at one offset). When a dispatch binds the buffer at offset `off`, `encode()` / `run_icb()` pick chunk `min(off / stride, n - 1)` and bind it at `off - k * stride`, so every bound range lies whole inside its chunk.
- `metal/src/engine/window.cpp`, `prompt_chain.inc`: the routed-expert kernels (decode `moe_gu_*`/`moe_down_*`, the prompt GEMMs and the prompt chain) bound the model at offset 0 and passed the experts' absolute file offsets in their argument structs. They are now bound at the layer's first expert tensor with offsets relative to it (`moe_rebase`). `max_span` = the largest tensor or the largest layer's gate+up+down range (1.20 GB here).
## What it leaves alone
- A model that fits one buffer (every pack on a 128 GB Mac) takes the old `wrap(ptr, bytes)` path, and `moe_rebase` returns 0 there, so the default path is unchanged.
- The automatic resident/stream decision (`ExpertStream::wanted`, setup) is unchanged: a 64 GB Mac still streams unless `STRATA_METAL_STREAM=0` is set. I did not change the default because I have not checked the memory headroom with other programs running beside it.
- No CUDA / HIP / SYCL file is touched. `STRATA_METAL_WRAP_SPLIT=0` keeps the old single-buffer wrap.
## How I checked
Mac Studio M4 Max 64 GB (macOS 15), Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, same build otherwise:
- `strata generate` with a 12-token and a 360-token prompt: resident (`STRATA_METAL_STREAM=0`) and streamed give the same tokens, and the traced top logits are equal.
- Through the server (greedy, `reasoning_effort: none`): the 400-token code and story answers are identical resident vs streamed, and resident+MTP vs resident.
## Measured (same Mac, same day, decode tok/s)
`bench-strata.py --lengths 1024,4096 --max-tokens 256 --repeat 3` (1K / 4K), plus 400 greedy tokens of "Write a Python function that parses an ISO-8601 date string and returns the weekday name. Include error handling." (code) and "Write a short story about a lighthouse keeper who finds a message in a bottle." (story):
| | 1K | 4K | code | story |
|---|---:|---:|---:|---:|
| v0.1.44, setup's config for 64 GB (experts streamed, no MTP) | 52.6 | 52.0 | 29.1 | 37.1 |
| this PR, resident, no MTP | 59.7 | 59.0 | 58.1 | 58.3 |
| this PR, resident, MTP draft layer, `--spec 2` | 73.5 | 72.3 | 69.5 | 59.3 |
(`--spec 3`: 67.0 / 70.7 / 67.1 / 48.7; `--spec 4`: 73.9 / 72.9 / 67.3 / 41.5; suffix drafter on: 73.7 / 71.9 / 69.8 / 59.1.)
## Not tested
Other Macs (M1-M3, M5, 96 GB), Q2_0 / IQ3_S packs, memory pressure with other large programs open, and contexts beyond 32K.
関連リンク
インストール・モデル・リリースへの站内リンク。