#5995 [Engine] Exit code 0xC0000409: stack buffer overrun on V100 + RTX A4500
closed · @pylorak · 4 comments · View on GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Description
I have two GPUs, a V100 32GB and an RTX A4500 20GB, with 64GB system RAM.
Using each GPU alone (e.g. specifying either --gpu 0, or --gpu 1) works perfectly.
But when I tell Strata to use both GPUs, it fails. Tried to switch roles (main/helper) between GPUs but no difference.
Windows 11, Strata engine 0.1.59, freshly updated with UPDATE.bat.
Logs:
c:\data\SD\Strata>.\START-HERE.bat --family unsloth --model UD-Q4_K_XL --vision cpu --gpu-roles 1=main,0=helper --cuda 12 --vram-reserve-mib 1024
Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)
=== Step 1: checking your PC ===
[!] older NVIDIA GPUs (Pascal / Volta) can be used (--cuda 12): experimental, through a second engine built with CUDA 12 (docs/OLDER_GPUS.md)
Your NVIDIA GPUs:
GPU 0: NVIDIA RTX A4500, 20 GB VRAM - can be used
GPU 1: Tesla V100-PCIE-32GB, 32 GB VRAM - can be used
[ok] GPUs: GPU 0 (NVIDIA RTX A4500, 20 GB) + GPU 1 (Tesla V100-PCIE-32GB, 32 GB) together (the model's layers are split across them)
[ok] GPU: Tesla V100-PCIE-32GB, 32.0 GB VRAM, compute capability 7.0, driver 582.99
[!] CUDA 12: --cuda 12 (as you chose)
[ok] RAM: 64 GB
[ok] CPU: AMD Ryzen 9 7900 12-Core Processor (AVX-512)
[ok] PCIe: 3.0 x4 - up to ~4 GB/s to the GPU: long prompts are read slower than on a PCIe 4.0 x16 PC (their experts are copied over it); the engine measures the link at start for the output speed
[!] the GPU's PCIe link reads 4 of its 16 lanes right now (some cards narrow it when idle); if it stays narrow under load: is it in the right slot (the one wired x16), fully seated, and not sharing lanes with an M.2 drive? (the BIOS can say)
=== Step 2: your choices ===
[ok] model: Qwen3.8-Flash-Next (Unsloth)
1) UD-IQ4_XS (recommended) ~4-bit i-quant (Unsloth Dynamic), between IQ3_S and UD-Q4_K_XL in quality; on a PC with less than ~80 GB of RAM part of its experts are read from the SSD; download 94 GB, keeps ~40 GB of its 60 GB of experts in RAM <- fits
2) UD-Q4_K_XL (recommended, installed) 4-bit (Unsloth Dynamic): K-quant experts, between UD-IQ4_XS and UD-Q5_K_XL in quality; download 111 GB, keeps ~40 GB of its 77 GB of experts in RAM <- fits
3) UD-Q5_K_XL 5-bit (Unsloth Dynamic): Q5_K experts with Q8_0 downs, close to Q8_0 in quality; download 158 GB, keeps ~40 GB of its 98 GB of experts in RAM <- may run slower
4) UD-Q6_K_XL 6-bit (Unsloth Dynamic): Q6_K experts with Q8_0 downs; download 169 GB, keeps ~40 GB of its 109 GB of experts in RAM <- may run slower
5) Q8_0 8-bit, the reference quality (every expert Q8_0); for a 192 GB PC; download 188 GB, keeps ~40 GB of its 128 GB of experts in RAM <- may run slower
6) UD-IQ1_M ~1-bit (Unsloth Dynamic v3.1), the smallest download; download 68 GB, keeps all of its 35 GB of experts in RAM <- fits
7) UD-Q2_0 2-bit (Unsloth Dynamic v3.1); download 70 GB, keeps all of its 37 GB of experts in RAM <- fits
8) UD-IQ2_XS 2-bit i-quant (Unsloth Dynamic v3.1); download 76 GB, keeps all of its 42 GB of experts in RAM <- fits
9) UD-Q2_K_XL 2-bit (Unsloth Dynamic v3.1) with some 3-bit experts; download 80 GB, keeps all of its 46 GB of experts in RAM <- fits
10) UD-IQ3_XXS 3-bit i-quant (Unsloth Dynamic v3.1); download 84 GB, keeps all of its 50 GB of experts in RAM <- fits
11) UD-Q3_K_XL 3-bit (Unsloth Dynamic v3.1) with some 4-bit experts; download 90 GB, keeps ~40 GB of its 56 GB of experts in RAM <- fits
12) UD-IQ4_XS-v31 ~4-bit i-quant (Unsloth Dynamic v3.1), a newer file than UD-IQ4_XS with other tensor types; download 94 GB, keeps ~40 GB of its 60 GB of experts in RAM <- fits
[ok] UD-Q4_K_XL on GPU 0 (NVIDIA RTX A4500, 20 GB) + GPU 1 (Tesla V100-PCIE-32GB, 32 GB), as you chose (--gpus), with its RAM budget (each card caches its own layers' experts, the RAM the hottest of the rest; #642)
[ok] RAM budget: 40 GiB of UD-Q4_K_XL's experts in RAM, the rest read from the model files on the SSD
[ok] size: UD-Q4_K_XL
Context length = how much text the model can see at once (your chat, files, tool output).
Longer needs more VRAM for it, so fewer experts fit on the GPU:
1) 8K tokens
2) 32K tokens
3) 64K tokens
4) 128K tokens (recommended for your GPU)
5) 200K tokens
6) 256K tokens
7) 384K tokens (experimental: setup adds rope scaling)
8) 512K tokens (experimental: setup adds rope scaling)
Context? [4]: 6
[ok] context: 262144 tokens
KV cache precision (the model's memory of the conversation):
1) 8-bit (recommended: what every published number was measured with)
2) 4-bit half the memory (about 4% faster at 128K), but measurably less precise on long
documents; long-context lookups (needle tests) still pass
KV cache? [1]: 1
[ok] KV cache: 8-bit
[ok] images: on (encoder on the CPU)
=== Step 3: Python packages ===
[ok] numpy, jinja2, regex, pyyaml, tqdm, requests, pillow, psutil, textual (the screens) already installed
=== Step 4: the Strata engine ===
[ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
[ok] engine already built for this PC
[ok] engine: C:\data\SD\Strata\engine-cuda12\strata.exe
=== Step 5: downloading Qwen3.8-Flash-Next (Unsloth) UD-Q4_K_XL ===
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf already downloaded
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf already downloaded
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf already downloaded
[ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf already downloaded
[ok] model files present
[ok] vision encoder already downloaded
[ok] vision encoder: C:\data\SD\Strata-data\models\mmproj-Qwen3.8-Flash-Next-BF16.gguf
tip: on the CPU a Q8_0 copy of this encoder is a third smaller and about as exact (embedding cosine 0.999 vs BF16); see 'A Q8_0 encoder' in docs/DETAILS.md
=== Step 6: preparing the model for Strata ===
[ok] model prepared: C:\data\SD\Strata-data\packs\unsloth-ud-q4_k_xl
[ok] MTP draft layer: C:\data\SD\Strata-data\mtp\rt
=== Step 7: writing the start script ===
[ok] KV streaming on: the context's KV cache lives in RAM (3.6 GB), more experts fit in VRAM
[ok] RAM budget: 36 GiB (less the KV cache's RAM)
[ok] VRAM kept free for other programs: 1024 MiB (--vram-reserve-mib; the expert cache takes that much less)
[ok] layer split across GPUs [0, 1] (auto)
[ok] kept from your earlier strata-unsloth-ud-q4_k_xl.json: user_args, args --expert-cache, args --spec-min-p, args --vram-reserve-mib
the earlier run config is kept as strata-unsloth-ud-q4_k_xl.json.bak
Tune Strata for this PC now? It measures a few engine settings (about 15-30 minutes, longer on a slow card; the PC is busy meanwhile; later: START-HERE --calibrate) [y]:
Tuning Strata for this PC: the output speed is measured with a few engine settings (the PCIe share, the
draft depth, the CPU threads, the expert cache; on two or more cards also the prompt chunks). It takes
roughly 15-30 minutes (3 rounds of each value); the PC is busy meanwhile.
[strata] drafting: DFlash2 alone (--draft dflash2, --draft-conf 0.35)
Loading the model for the measurements ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 36 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the server says when it is ready.
[strata] filling the GPU's expert cache (3998 experts, 11.66 GiB of VRAM) ...
[strata] almost ready ...
[!] the tuning did not finish (the engine exited before it was ready (see C:\data\SD\Strata\strata-unsloth-ud-q4_k_xl.log)
the engine process ended: exit code 0xC0000409: stack buffer overrun / fast-fail
the engine log's last lines:
FileExpertSource: copied cache complement through layer 24/48 (26.44 GiB, 5 s)
FileExpertSource: copied cache complement through layer 32/48 (31.80 GiB, 6 s)
FileExpertSource: copied cache complement through layer 40/48 (31.80 GiB, 6 s)
FileExpertSource: copied cache complement through layer 48/48 (34.33 GiB, 6 s)
FileExpertSource: mapped pinned cache complement ready: resident 34.33 GiB, pinned 34.33 GiB in 11 s; page-locked and mapped
FileExpertSource: the split stages lend 2546 slots to the prompt path: 0 keep their experts in RAM too (0.00 GiB) (the rest are read from the file when lent: not enough RAM for them)
FileExpertSource: 7931 verified additional-GPU experts remain on the mmap fallback
strata generate: WARNING: a prompt chunk of 8192 tokens lends 1276 cache slots to the prompt path, but only 0 of their experts fit in RAM: the others are read from the SSD on every chunk, and prompts can read ~3x slower than with --prefill auto (#669). A larger --resident-budget-gib (the lend region comes after the ranked experts), or a smaller --prefill, keeps them all in RAM
strata generate: resident RAM mode: 34.33 GiB of experts in RAM (page-locked), 3998 in the GPU cache; adaptive swaps exchange them with the GPU cache (no file reads)
strata generate: the file tier reads unbuffered (re-checked with the RAM copy built (34.33 GiB): 13.5 GiB available, 0.0 GiB of it still to be taken by the RAM copy, 37.4 GiB of experts read from the files: the file cache cannot keep them)
strata generate: the mapped view of the experts is closed (3 GGUF shards unmapped; every expert outside RAM and VRAM is read unbuffered)
strata serve: Windows commit: 3.7 GiB left of a 79.7 GiB limit (RAM + page file)
strata serve: WARNING the Windows commit limit is 79.7 GiB (RAM + page file) and 3.7 GiB of it is left, but this run plans 2.0 GiB of host allocations and keeps 4.0 GiB free: it can end at the limit with no message (exit code 0xC0000409, #1607) while RAM is still free. Raise the page file (System managed, or RAM-sized) or lower --conversation-cache-mib / --max-context / --prefill
strata: prefill plan: chunk 8192, ring 96 kept: file set 10.4 GiB does not fit the page cache 0.0 GiB -> about 44.6 GB read from the drive per 32768-token prompt (4 chunks); no option does better (the set cannot fit: fewest passes win)
strata: prefill plan inputs: 71.7 GiB experts, 34.8 GiB in the GPU caches, 34.3 GiB RAM copy (0 lend slots kept), file tier 2.6 GiB; 0.0 GiB available for the page cache (the file tier reads unbuffered: none); PCIe 3.2 GB/s; drive 1.50 GB/s (assumed); a 32768-token prompt
strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
strata serve: the prompt path borrows 1276 CUDA0 cache slots (3.73 GiB)
strata serve: CUDA1 prompt path borrows 1270 of its 7931 slots (3.73 GiB)
strata serve: lend sizing: the prompt path borrows 2546 slots (ring 96 slots at chunk 8192), 0 of them keep their experts in RAM too
strata serve: WARNING: 2546 of the 2546 borrowed slots have no RAM copy: their experts are streamed from the model file during every chunk): the default settings stay
the engine said: strata serve: WARNING: 2546 of the 2546 borrowed slots have no RAM copy: their experts are streamed from the model file during every chunk
[ok] start script: run-unsloth-ud-q4_k_xl.bat
All set.
API (OpenAI): http://127.0.0.1:8080/v1 (any API key; model name: anything)
API (Anthropic): http://127.0.0.1:8080/v1/messages
Next time: just run START-HERE.bat (or run-unsloth-ud-q4_k_xl.bat) - it starts right away
Images: send them in the chat page, in chat.py (/image <path>) or over the API
Several requests at once (opt-in): --parallel 4 decodes up to 4 together instead of one after the other (each takes ~0.6 GB of VRAM from the expert cache; docs/BATCHING.md).
Tuning: FAILED (the reason is above): the default settings stay - START-HERE.bat --calibrate tries again
Run a quick benchmark now? It measures this install for about 10 minutes and writes a report you can share (later: BENCH.bat) [n]:
stopped.
Terminate batch job (Y/N)?
^CPRelated on this site
Quick links to related pages.