Pull requests / #3600
#3600 [Engine] Metal: Context::free gives the buffer back (alloc and wrap drain their own autorelease pool)
open · @Flo5k5 · 0 comments · View on GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quants
Description
## Problem and result On the Mac engine, `mtl::Context::free` releases the buffer's handle but the memory stays allocated for the life of the process. Metal keeps an autoreleased reference to every new `MTLBuffer`, and the engine's threads have no autorelease pool, so that reference is never dropped. `Context::run` and the other submit paths already wrap their work in `@autoreleasepool`; `alloc` and `wrap` did not. Measured on an Apple M5 Pro: 40 buffers of 100 MB allocated, used by one dispatch and freed, four times over, leave **16.8 GB** of process footprint (`task_vm_info.phys_footprint`) on `main`. With this change: **0.17 GB** after each free, and no growth from one round to the next. This is what makes #3527 effective. That PR frees the prompt chain's old buffer set when `pc_setup` grows (`prompt_chain.inc:60`), but on `main` those `free` calls give nothing back. With Qwen3.8-Flash-Next Q2_0 and an agent client whose first prompt per task is 2,004-2,026 tokens (just under the 2,048-row chunk), each new task grows the set by a few rows and loses about 4 GB. Engine footprint (`vmmap -summary`, physical footprint) after fresh conversations of 2,004, 2,017 and 2,600 tokens, then one of 5,279 tokens and its second turn: | Build | Sets allocated (rows) | Footprint after each prompt | |---|---|---| | `main` | 1,997, 2,010 | 10.5 → 14.3 GB (run stopped there) | | `main` + the #3527 change resolved onto `main` (64K context, prompts of 1,963 and 1,973 tokens) | 1,956, 1,966 | 11.8 → 15.5 GB, as `main`; an external memory watchdog stopped the third prompt (swap +3.8 GB) | | `main` + the #3527 change + this PR | 1,997, 2,010, 2,048 | 10.5 → 10.5 → 10.6 → 10.7 GB | ## Related discussion #3527 (the `pc_setup` side of the same leak, open). It and this PR are independent: this one makes `free` work; #3527 calls it where it was missing. Neither is enough alone for the prompt chain. Its model-free test passes because the test drains autorelease pools itself before checking the buffers' lifetime (which is also why it does not see this). As of 011c9e60 it no longer compiles against `main`: the extracted `pc_setup` now calls `T(...)` and reads `had_on_` (Gyro). ## Approach - `metal/src/mtl_rt.mm`: `Context::alloc`, `Context::wrap(p, bytes)` and each chunk of `Context::wrap(p, bytes, max_span)` run inside `@autoreleasepool`, so the only reference that remains is the one the `Buffer` handle holds (`__bridge_retained`), and `free` drops the last one. No behaviour change for buffers that are never freed. - `metal/tests/buffer_free_test.cpp`: the check above, on a thread with no pool (as the engine's are). It does not need a model. - Callers of `Context::free` in the engine today: the `coop_k4` probe buffer (once, a few KB) and `mpemb_` in `mtp_prefill` (at most ~21 MB per growth). The prompt chain gets one with #3527. ## Validation Revision: `fix/metal-free-autorelease` at b19df7bb (main 011c9e60 + this change). MacBook Pro, Apple M5 Pro, 64 GB, macOS 27. `cmake -S metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=… -DSTRATA_GGML_DIR=…` (llama.cpp 3cf03257), `ninja -C build-metal`. **Passed:** - Build: every target links; no new warning. - `build-metal/buffer_free_test`: PASS (9 checks). The same source linked against `main`'s `libmtl_rt.a`: FAILED (16.85 GB held after the fourth free), as expected. - `metal/tests`: 20 of 20 programs pass (runtime_hello, arena_budget_test, wrap_split_test, coop_layout, dense_mm, expert, fuse, gdn, gdn_prompt, gemv, gr2, gr, mma, moe_mm, pair, prompt, qsa, qsa_prompt, route and sample parity), no FAIL line. - Engine, Qwen3.8-Flash-Next Q2_0 (`--spec 2 --mtp … --prefill auto --max-context 32768 --mmap-experts`, `STRATA_METAL_STREAM=0`, `STRATA_METAL_PROMPT_CHUNK=2048`), served by `serve/server.py`, with the #3527 change on top: the footprint table above. The `IOAccelerator (graphics)` region stays at 9.7-9.8 GB over the whole sequence (main: 9.7 → 13.5 GB after the second prompt). Each growth of the set takes 192-319 ms. - Greedy replies, a fresh server per build, `main` against `main` + #3527 + this PR: 7 of 7 identical, character for character (5 short prompts, a 6,468-token prompt read in 4 chunks, its second turn). **Failed:** - none observed **Not run:** - CUDA, HIP and SYCL builds: no file they compile is changed. - Other Macs (M4, M5 Max). - The SSD-streaming path (`STRATA_METAL_STREAM=1`) as a whole; its buffers are allocated once and are not freed. - A long agent run without warm-up: one was stopped after 129 s by the external memory watchdog with the Mac already holding 9.5 GB of compressed memory from other work before the engine started; the engine itself allocated three sets and its footprint went 8.0 → 12.7 GB (one set plus checkpoints). ## Limits - `Context::allocated()` still counts every allocation since start (unchanged; it never subtracted freed bytes). - Other Objective-C objects created outside a pool (pipelines, libraries) are made once and kept, so they are left as they are.
Related on this site
Quick links to related pages.