#6447 [Engine] Size Metal state after batch admission

open · @Maryarty · 0 评论 · 去 GitHub 看

Server & APIModels & quants

说明

## Problem and result

Metal sized its decode state from the requested `--batch` before the host validated the number of slots. For example, the host disables `--batch 1`, which requires at least two slots, but Metal still allocated an extra set of GDN and QSA state at startup. With a 16,384-token cap on M1 Max, the streaming engine reported 4.65 GB before the arena with batch off and 5.00 GB with the rejected slot. Both settings report 4.65 GB after the fix.

## Related discussion

None. This fixes the timing of an existing Metal-only configuration call.

## Approach

Pass the admitted slot count to Metal once the host has applied its batch and memory limits. `Window::init` can then allocate the solo state and one set per admitted slot. The changed statements are inside the existing Metal compile guard, and the batch admission rules stay in the host code.

## Validation

**Passed**

- Release arm64 Metal build at `1040719a85119a420b4c75dc00e520a09d88b8ee` from main `3142b66`. `ctest --test-dir engine/build-metal -R '^runtime_hello$' --output-on-failure` passed on M1 Max, 32 GPU cores, 64 GiB, macOS 26.6.2 (25G83).
- Four `strata --serve` runs, main and patch with batch off and with `--batch 1`. Qwen3.8-Flash-Next GSQ-RCO Q2_0 with PLE, `STRATA_METAL_STREAM=1`, `STRATA_METAL_ARENA_GB=12`, `--max-context 16384`, `--spec 2`. Each used the same 16 fixed input IDs and generated 128 greedy IDs. `STRATA_DUMP_FIRST_LOGITS` matched byte for byte in all four runs. Output IDs and MTP acceptance (63/64) matched, and all runs had zero new Swapouts.
- On the patch, `strata --serve --batch 2` admitted and finished two eight-token requests (`BGEN 0 8` and `BGEN 1 8`) with `BDONE ... 8 length` on both slots.

**Failed**

Main reproduced the extra allocation. None observed after the change in the above checks.

**Not run**

Other Apple GPU generations and other backends. This diff changes code inside `STRATA_GPU_METAL` guards only.

## Limits

The 0.35 GB change is the engine's Metal allocation counter at one context length and phase. The per-slot buffer size scales with context. Accepted batch slots still need their own state sets.

本站相关内容

相关页面的快捷入口。