Issues / #828

#828 Heap corruption (munmap_chunk: invalid pointer) at startup in Prefill::init — crashes with 1/2/4 GPUs

open · @wehooper4 · 3 comments · View on GitHub

Server & APIMulti-GPUNVIDIA / CUDAModels & quants

Description

## Crash
Engine crashes at startup with `munmap_chunk(): invalid pointer` (heap corruption) during prompt path initialization. Happens consistently, 100% reproducible.

## Log (last lines before crash)
```
strata generate: layer split: CUDA0 runs layers 0-11
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 8 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.39)
strata generate: token graph hit path: 16530 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
munmap_chunk(): invalid pointer
```

With default settings (no STRATA_SPLIT_OWN), it crashes at:
```
strata serve:   CUDA3 prompt path borrows 2914 of its 4526 slots (4.03 GiB)
munmap_chunk(): invalid pointer
```

## System
- 4x RTX 3060 12GB (sm_86)
- 128GB DDR4-3200
- Debian 13 (trixie), glibc 2.41
- CUDA 13.0 toolkit, NVIDIA driver 595.91.07
- Engine 0.1.39 (commit 6f32ec0, latest main as of 2026-10-04)
- Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS, 262144 context, --vision yes
- Running in LXC container

## What I've tried
- 1, 2, and 4 GPUs (--gpus 0 / 0,1 / all) — all crash identically
- With and without STRATA_SPLIT_OWN=1 — both crash
- STRATA_RING_BYTES=0 (opt out of #583) — still crashes
- 32k vs 262k context — both crash
- Increased --vram-reserve-mib to 2000 — still crashes
- MALLOC_CHECK_=3 confirms heap corruption (process aborts instead of munmap error)

## Analysis
The crash is inside `Prefill::init()` (src/prefill/prefill.cpp). The `borrow` buffer (from `ExpertCache::device_slot()`) is valid — I verified the slot index is in-bounds and the pointer is non-null. The corruption happens after `Prefill::init` receives the borrow buffer.

The #583 "byte-budget ring" changes are the likely culprit (all recent changes to prefill.cpp are from #583), but `STRATA_RING_BYTES=0` does not fully disable the code path.

This is different from Issue #70 (which crashes mid-decode after successfully serving). This crashes at startup, before serving any requests.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.