#6656 [Engine] inkling2 Metal: bind the experts from their own buffer when the model exceeds maxBufferLength

open · @dailytrap · 0 Kommentare · Auf GitHub

Server & APINVIDIA / CUDAModels & quants

Beschreibung

## Problem and result
On a Mac whose `maxBufferLength` is smaller than the `.strata` file, Inkling-2-Mini answers only `!` (token id 0) with `finish_reason: "length"` (#6630). Reproduced here on an Apple M5, 24 GB, `maxBufferLength` 14.30 GB, with SQ-15 (14.35 GB).

Cause: the expert kernels bind Section 4 once (`sec4_arg()` → `Arg::B(wbuf, sec4_off)`) and add each routed expert's `body_off` inside the kernel. The generic chunked `Context::wrap(p, bytes, max_span)` resolves that binding to chunk 0, which ends at `maxBufferLength`. Everything in Section 4 past that point is outside the bound `MTLBuffer`. For SQ-15 that is 13 experts of physical layer 23 (file offsets 14 293 934 080 … 14 348 193 792, read from the Section 3 directory). Once the router picks one, the logits degenerate (all −30 after the softcap) and argmax is 0.

After the change the same model, machine and request answer correctly.

## Related discussion
Fixes #6630. Small, local bug fix; no scope change.

## Approach
When the file is larger than `maxBufferLength`, `InkMetalModel::load` now wraps it with the existing `Context::wrap_ranges` as two page-aligned buffers:
- `[0, roundup(sec4_off))`: the header and Section 1 (trunk, dspark), which `tw()` / `dw()` bind tensor by tensor
- `[rounddown(sec4_off), map_len)`: Section 4 to the end of the file, so `sec4_arg()` reaches every expert from one binding

The two ranges overlap by less than one page, and `BO()` picks the range by offset. If either range is still larger than `maxBufferLength` (for example SQ-20 on a 24 GB Mac: Section 4 ≈ 17 GB), the load now fails with a message that names the sizes, where it used to load and then produce garbage. That case was already broken before this change, so nothing that works today stops working. Files that fit one buffer still take the unchanged single `wrap()` path.

## Validation
Revision: `c6f6427` + this commit. Apple M5, 24 GB unified memory, `maxBufferLength` 14.30 GB, model `Inkling-2-Mini-SQ-15.strata` (14 348 316 672 bytes).

**Passed:**
- `engine/src/inkling2/metal/build_mac.sh`, then `strata-inkling-metal --native Inkling-2-Mini-SQ-15.strata --max-context 4096 --selfcheck`
  - without the change: `mtl: wrapped 14.35 GB as 2 overlapping buffers …` → `[SELFCHECK] P=8 … nan=0 top1=0 | top5: 2(-30.0000) 4(-30.0000) 0(-30.0000) 1(-30.0000) 3(-30.0000)`
  - with the change: `mtl: wrapped 14.35 GB as 2 separate ranges (of 14.35 GB)` → `[SELFCHECK] P=8 … nan=0 top1=198 | top5: 198(24.9504) 68014(24.7897) 220(24.5413) 11(23.9222) 59(23.4000)`
- CMake/Ninja build of the `strata-inkling` target, served through `serve.server` (control 0.1.65) with the launch config `sq-15-32k` (32K context, vision on, `--draft dspark`). `POST /v1/chat/completions` with `"Sag Hallo und nenne die Hauptstadt von Frankreich."`, `temperature: 0`:
  - released engine 0.1.65: `"!!!!\n\n!!!!\n\n!!!!!!…"`, `finish_reason: "length"`
  - with the change: `"Hallo! Die Hauptstadt von Frankreich ist Paris."`, `finish_reason: "stop"`

**Failed:**
none observed

**Not run:**
- Macs whose `maxBufferLength` covers the whole file. The code path there is unchanged.
- SQ-20 on a 24 GB Mac (the new clear-error path). I checked it only by reading the code: Section 4 alone is larger than `maxBufferLength` there.
- Vision requests. The vision file (636 MB) uses its own unchanged single `wrap()`.
- Long-context or performance runs. The kernels and dispatches are unchanged; only which `MTLBuffer` and offset each binding resolves to has changed.

## Limits
A model whose Section 4 alone is larger than `maxBufferLength` (SQ-20 on 24 GB Macs) still cannot run. Supporting it would need the expert kernels to take several buffers, or split experts by chunk. This PR makes that case fail at load with an explanation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.