Pull requests / #1283

#1283 prefill: bytes_needed sizes the MoE buffers for the layout init picks (with or without an expert source)

open · @sergqwer · 0 コメント · GitHub で見る

Setup & install

本文

Replaces #1042, which GitHub closed when `main`'s history was rewritten. The closing note says this was not a rejection. It is the same commit, cherry-picked onto the new `main` (v0.1.40.1).

---

`Prefill::bytes_needed()` sizes the region the prompt path borrows from the expert cache. It must count exactly what `carve()` allocates. 0.1.39 took the first part of #547 with #583 (the `xn` / `grs` count). This is what is left.

**The MoE layout.** `bytes_needed` sizes the MoE scratch with `fused_layout(T, true)` for every caller. `carve` takes the fused layout only when `init` gets an `ExpertSource` (`fused_layout(T, m.src != nullptr)`). So a prompt path without one is lent the fused layout's smaller MoE buffers, and its unfused buffers do not fit in them.

**The change.** `bytes_needed_impl` takes a `src` flag. So do `bytes_needed`, `bytes_needed_no_ring` and 0.1.40's `bytes_needed_owned` (#796), all defaulting to `true`. Every caller in `generate.cpp` passes `srcp != nullptr`, the same source it hands `init`. That includes `STRATA_OWNED_PRICE=exact`'s sizing.

**Identity.** With an `ExpertSource` (every normal run) nothing changes. The logits and the 32 greedy tokens after a 32K prompt equal v0.1.40 on IQ2_XS: identical bytes, same expert cache. The engine's sources here are the same bytes as on that branch, because 0.1.40.1 changed only the server and setup.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

関連リンク

インストール・モデル・リリースへの站内リンク。