Pull requests / #1454
#1454 prefill: reuse embedding storage for half outputs
open · @W1nge · 0 comentários · No GitHub
Descrição
Reuse the prompt embedding allocation for the later half output (`bo`). Embedding gathers, image-row uploads, the last embedding consumer (`gr_broadcast`), output projections, MoE combine and HC writes are ordered on `m.cs`. A following chunk reloads its embeddings; a stage receiving `hand_in_` does not read `emb`. PLE embeddings retain their separate allocation. Remove the corresponding `bo` allocation from `bytes_needed_impl` as well as `carve`. This saves `chunk * 2560 * sizeof(float)` payload bytes (320 MiB at 32768 rows), with allocator rounding handled by the existing allocator. No new flags, default chunk changes, or numerical kernels. Independent of #1451. Validation: reviewed all `m.emb`/`m.bo` accesses in current main d5ea713; `prefill.cpp` compiles with MSVC 19.44/CUDA 12.4 headers both without native/MMQ and with native/MMQ/fused enabled; `git diff --check` passes. Draft: this alias was part of the locally validated 32K fork, but this isolated upstream port has not had a full-model, peer/stage or other-backend runtime comparison. The allocation saving is not an isolated throughput measurement. No broader HC scratch overlay is included. Validation update (2026-10-08): a complete current-main integration build including #1451, #1454, #1455, #1457 and #1465 passed four real IQ3_XXS model baseline/integration pairs on RTX 2080 Ti: default, GR_UNFUSED, PREFILL_BF16X2 and legacy RING_BYTES modes. Each used INT8 KV, 194 input tokens, two prefill chunks and 8 output tokens; all runs exited successfully and output IDs matched within each pair. A 256K configured-context startup and short generation also passed (this did not fill a 256K prompt). These are combined regression checks, not an isolated performance result or exhaustive state equality. Other hardware/backends remain untested. The earlier draft-only status reflected the absence of any current-main model run; now requesting review with these limits explicit.
No site
Links install, modelos, releases.