Pull requests / #1283
#1283 prefill: bytes_needed sizes the MoE buffers for the layout init picks (with or without an expert source)
open · @sergqwer · 0 comentários · No GitHub
Descrição
Replaces #1042, which GitHub closed when `main`'s history was rewritten. The closing note says this was not a rejection. It is the same commit, cherry-picked onto the new `main` (v0.1.40.1). --- `Prefill::bytes_needed()` sizes the region the prompt path borrows from the expert cache. It must count exactly what `carve()` allocates. 0.1.39 took the first part of #547 with #583 (the `xn` / `grs` count). This is what is left. **The MoE layout.** `bytes_needed` sizes the MoE scratch with `fused_layout(T, true)` for every caller. `carve` takes the fused layout only when `init` gets an `ExpertSource` (`fused_layout(T, m.src != nullptr)`). So a prompt path without one is lent the fused layout's smaller MoE buffers, and its unfused buffers do not fit in them. **The change.** `bytes_needed_impl` takes a `src` flag. So do `bytes_needed`, `bytes_needed_no_ring` and 0.1.40's `bytes_needed_owned` (#796), all defaulting to `true`. Every caller in `generate.cpp` passes `srcp != nullptr`, the same source it hands `init`. That includes `STRATA_OWNED_PRICE=exact`'s sizing. **Identity.** With an `ExpertSource` (every normal run) nothing changes. The logits and the 32 greedy tokens after a 32K prompt equal v0.1.40 on IQ2_XS: identical bytes, same expert cache. The engine's sources here are the same bytes as on that branch, because 0.1.40.1 changed only the server and setup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
No site
Links install, modelos, releases.