Issues / #828
#828 Heap corruption (munmap_chunk: invalid pointer) at startup in Prefill::init — crashes with 1/2/4 GPUs
open · @wehooper4 · 3 评论 · 在 GitHub 查看
Server & APIMulti-GPUNVIDIA / CUDAModels & quants
描述
## Crash Engine crashes at startup with `munmap_chunk(): invalid pointer` (heap corruption) during prompt path initialization. Happens consistently, 100% reproducible. ## Log (last lines before crash) ``` strata generate: layer split: CUDA0 runs layers 0-11 strata generate: R4 hit path ON - resident experts are computed on the GPU strata generate: 8 expert-pool workers + the host thread strata generate: session is up (engine 0.1.39) strata generate: token graph hit path: 16530 resident experts, decided on the device strata serve: the prompt path allocates its own buffers (too few cache slots to borrow) munmap_chunk(): invalid pointer ``` With default settings (no STRATA_SPLIT_OWN), it crashes at: ``` strata serve: CUDA3 prompt path borrows 2914 of its 4526 slots (4.03 GiB) munmap_chunk(): invalid pointer ``` ## System - 4x RTX 3060 12GB (sm_86) - 128GB DDR4-3200 - Debian 13 (trixie), glibc 2.41 - CUDA 13.0 toolkit, NVIDIA driver 595.91.07 - Engine 0.1.39 (commit 6f32ec0, latest main as of 2026-10-04) - Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS, 262144 context, --vision yes - Running in LXC container ## What I've tried - 1, 2, and 4 GPUs (--gpus 0 / 0,1 / all) — all crash identically - With and without STRATA_SPLIT_OWN=1 — both crash - STRATA_RING_BYTES=0 (opt out of #583) — still crashes - 32k vs 262k context — both crash - Increased --vram-reserve-mib to 2000 — still crashes - MALLOC_CHECK_=3 confirms heap corruption (process aborts instead of munmap error) ## Analysis The crash is inside `Prefill::init()` (src/prefill/prefill.cpp). The `borrow` buffer (from `ExpertCache::device_slot()`) is valid — I verified the slot index is in-bounds and the pointer is non-null. The corruption happens after `Prefill::init` receives the borrow buffer. The #583 "byte-budget ring" changes are the likely culprit (all recent changes to prefill.cpp are from #583), but `STRATA_RING_BYTES=0` does not fully disable the code path. This is different from Issue #70 (which crashes mid-decode after successfully serving). This crashes at startup, before serving any requests.
站内延伸阅读
链到安装、模型与版本说明,便于 SEO/GEO,非官方 issue 正文。