#5995 [Engine] Exit code 0xC0000409: stack buffer overrun on V100 + RTX A4500

closed · @pylorak · 4 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Beschreibung

I have two GPUs, a V100 32GB and an RTX A4500 20GB, with 64GB system RAM.

Using each GPU alone (e.g. specifying either --gpu 0, or --gpu 1) works perfectly.
But when I tell Strata to use both GPUs, it fails. Tried to switch roles (main/helper) between GPUs but no difference. 

Windows 11, Strata engine 0.1.59, freshly updated with UPDATE.bat.

Logs:

c:\data\SD\Strata>.\START-HERE.bat --family unsloth --model UD-Q4_K_XL --vision cpu --gpu-roles 1=main,0=helper --cuda 12 --vram-reserve-mib 1024
Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)

=== Step 1: checking your PC ===
  [!]  older NVIDIA GPUs (Pascal / Volta) can be used (--cuda 12): experimental, through a second engine built with CUDA 12 (docs/OLDER_GPUS.md)
  Your NVIDIA GPUs:
    GPU 0: NVIDIA RTX A4500, 20 GB VRAM - can be used
    GPU 1: Tesla V100-PCIE-32GB, 32 GB VRAM - can be used
  [ok] GPUs: GPU 0 (NVIDIA RTX A4500, 20 GB) + GPU 1 (Tesla V100-PCIE-32GB, 32 GB) together (the model's layers are split across them)
  [ok] GPU: Tesla V100-PCIE-32GB, 32.0 GB VRAM, compute capability 7.0, driver 582.99
  [!]  CUDA 12: --cuda 12 (as you chose)
  [ok] RAM: 64 GB
  [ok] CPU: AMD Ryzen 9 7900 12-Core Processor (AVX-512)
  [ok] PCIe: 3.0 x4 - up to ~4 GB/s to the GPU: long prompts are read slower than on a PCIe 4.0 x16 PC (their experts are copied over it); the engine measures the link at start for the output speed
  [!]  the GPU's PCIe link reads 4 of its 16 lanes right now (some cards narrow it when idle); if it stays narrow under load: is it in the right slot (the one wired x16), fully seated, and not sharing lanes with an M.2 drive? (the BIOS can say)

=== Step 2: your choices ===
  [ok] model: Qwen3.8-Flash-Next (Unsloth)

  1) UD-IQ4_XS (recommended) ~4-bit i-quant (Unsloth Dynamic), between IQ3_S and UD-Q4_K_XL in quality; on a PC with less than ~80 GB of RAM part of its experts are read from the SSD; download 94 GB, keeps ~40 GB of its 60 GB of experts in RAM   <- fits
  2) UD-Q4_K_XL (recommended, installed) 4-bit (Unsloth Dynamic): K-quant experts, between UD-IQ4_XS and UD-Q5_K_XL in quality; download 111 GB, keeps ~40 GB of its 77 GB of experts in RAM   <- fits
  3) UD-Q5_K_XL 5-bit (Unsloth Dynamic): Q5_K experts with Q8_0 downs, close to Q8_0 in quality; download 158 GB, keeps ~40 GB of its 98 GB of experts in RAM   <- may run slower
  4) UD-Q6_K_XL 6-bit (Unsloth Dynamic): Q6_K experts with Q8_0 downs; download 169 GB, keeps ~40 GB of its 109 GB of experts in RAM   <- may run slower
  5) Q8_0 8-bit, the reference quality (every expert Q8_0); for a 192 GB PC; download 188 GB, keeps ~40 GB of its 128 GB of experts in RAM   <- may run slower
  6) UD-IQ1_M ~1-bit (Unsloth Dynamic v3.1), the smallest download; download 68 GB, keeps all of its 35 GB of experts in RAM   <- fits
  7) UD-Q2_0 2-bit (Unsloth Dynamic v3.1); download 70 GB, keeps all of its 37 GB of experts in RAM   <- fits
  8) UD-IQ2_XS 2-bit i-quant (Unsloth Dynamic v3.1); download 76 GB, keeps all of its 42 GB of experts in RAM   <- fits
  9) UD-Q2_K_XL 2-bit (Unsloth Dynamic v3.1) with some 3-bit experts; download 80 GB, keeps all of its 46 GB of experts in RAM   <- fits
  10) UD-IQ3_XXS 3-bit i-quant (Unsloth Dynamic v3.1); download 84 GB, keeps all of its 50 GB of experts in RAM   <- fits
  11) UD-Q3_K_XL 3-bit (Unsloth Dynamic v3.1) with some 4-bit experts; download 90 GB, keeps ~40 GB of its 56 GB of experts in RAM   <- fits
  12) UD-IQ4_XS-v31 ~4-bit i-quant (Unsloth Dynamic v3.1), a newer file than UD-IQ4_XS with other tensor types; download 94 GB, keeps ~40 GB of its 60 GB of experts in RAM   <- fits
  [ok] UD-Q4_K_XL on GPU 0 (NVIDIA RTX A4500, 20 GB) + GPU 1 (Tesla V100-PCIE-32GB, 32 GB), as you chose (--gpus), with its RAM budget (each card caches its own layers' experts, the RAM the hottest of the rest; #642)
  [ok] RAM budget: 40 GiB of UD-Q4_K_XL's experts in RAM, the rest read from the model files on the SSD
  [ok] size: UD-Q4_K_XL

  Context length = how much text the model can see at once (your chat, files, tool output).
  Longer needs more VRAM for it, so fewer experts fit on the GPU:
  1) 8K tokens
  2) 32K tokens
  3) 64K tokens
  4) 128K tokens   (recommended for your GPU)
  5) 200K tokens
  6) 256K tokens
  7) 384K tokens   (experimental: setup adds rope scaling)
  8) 512K tokens   (experimental: setup adds rope scaling)
Context? [4]: 6
  [ok] context: 262144 tokens

  KV cache precision (the model's memory of the conversation):
  1) 8-bit   (recommended: what every published number was measured with)
  2) 4-bit   half the memory (about 4% faster at 128K), but measurably less precise on long
             documents; long-context lookups (needle tests) still pass
KV cache? [1]: 1
  [ok] KV cache: 8-bit
  [ok] images: on (encoder on the CPU)

=== Step 3: Python packages ===
  [ok] numpy, jinja2, regex, pyyaml, tqdm, requests, pillow, psutil, textual (the screens) already installed

=== Step 4: the Strata engine ===
  [ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
  [ok] engine already built for this PC
  [ok] engine: C:\data\SD\Strata\engine-cuda12\strata.exe

=== Step 5: downloading Qwen3.8-Flash-Next (Unsloth) UD-Q4_K_XL ===
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf already downloaded
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf already downloaded
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf already downloaded
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf already downloaded
  [ok] model files present
  [ok] vision encoder already downloaded
  [ok] vision encoder: C:\data\SD\Strata-data\models\mmproj-Qwen3.8-Flash-Next-BF16.gguf
       tip: on the CPU a Q8_0 copy of this encoder is a third smaller and about as exact (embedding cosine 0.999 vs BF16); see 'A Q8_0 encoder' in docs/DETAILS.md

=== Step 6: preparing the model for Strata ===
  [ok] model prepared: C:\data\SD\Strata-data\packs\unsloth-ud-q4_k_xl
  [ok] MTP draft layer: C:\data\SD\Strata-data\mtp\rt

=== Step 7: writing the start script ===
  [ok] KV streaming on: the context's KV cache lives in RAM (3.6 GB), more experts fit in VRAM
  [ok] RAM budget: 36 GiB (less the KV cache's RAM)
  [ok] VRAM kept free for other programs: 1024 MiB (--vram-reserve-mib; the expert cache takes that much less)
  [ok] layer split across GPUs [0, 1] (auto)
  [ok] kept from your earlier strata-unsloth-ud-q4_k_xl.json: user_args, args --expert-cache, args --spec-min-p, args --vram-reserve-mib
  the earlier run config is kept as strata-unsloth-ud-q4_k_xl.json.bak
Tune Strata for this PC now? It measures a few engine settings (about 15-30 minutes, longer on a slow card; the PC is busy meanwhile; later: START-HERE --calibrate) [y]:

  Tuning Strata for this PC: the output speed is measured with a few engine settings (the PCIe share, the
  draft depth, the CPU threads, the expert cache; on two or more cards also the prompt chunks). It takes
  roughly 15-30 minutes (3 rounds of each value); the PC is busy meanwhile.
[strata] drafting: DFlash2 alone (--draft dflash2, --draft-conf 0.35)
  Loading the model for the measurements ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 36 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
         YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
         Please wait and don't close this window; the server says when it is ready.
[strata] filling the GPU's expert cache (3998 experts, 11.66 GiB of VRAM) ...
[strata] almost ready ...
  [!]  the tuning did not finish (the engine exited before it was ready (see C:\data\SD\Strata\strata-unsloth-ud-q4_k_xl.log)
the engine process ended: exit code 0xC0000409: stack buffer overrun / fast-fail
the engine log's last lines:
  FileExpertSource: copied cache complement through layer 24/48 (26.44 GiB, 5 s)
  FileExpertSource: copied cache complement through layer 32/48 (31.80 GiB, 6 s)
  FileExpertSource: copied cache complement through layer 40/48 (31.80 GiB, 6 s)
  FileExpertSource: copied cache complement through layer 48/48 (34.33 GiB, 6 s)
  FileExpertSource: mapped pinned cache complement ready: resident 34.33 GiB, pinned 34.33 GiB in 11 s; page-locked and mapped
  FileExpertSource: the split stages lend 2546 slots to the prompt path: 0 keep their experts in RAM too (0.00 GiB) (the rest are read from the file when lent: not enough RAM for them)
  FileExpertSource: 7931 verified additional-GPU experts remain on the mmap fallback
  strata generate: WARNING: a prompt chunk of 8192 tokens lends 1276 cache slots to the prompt path, but only 0 of their experts fit in RAM: the others are read from the SSD on every chunk, and prompts can read ~3x slower than with --prefill auto (#669). A larger --resident-budget-gib (the lend region comes after the ranked experts), or a smaller --prefill, keeps them all in RAM
  strata generate: resident RAM mode: 34.33 GiB of experts in RAM (page-locked), 3998 in the GPU cache; adaptive swaps exchange them with the GPU cache (no file reads)
  strata generate: the file tier reads unbuffered (re-checked with the RAM copy built (34.33 GiB): 13.5 GiB available, 0.0 GiB of it still to be taken by the RAM copy, 37.4 GiB of experts read from the files: the file cache cannot keep them)
  strata generate: the mapped view of the experts is closed (3 GGUF shards unmapped; every expert outside RAM and VRAM is read unbuffered)
  strata serve: Windows commit: 3.7 GiB left of a 79.7 GiB limit (RAM + page file)
  strata serve: WARNING the Windows commit limit is 79.7 GiB (RAM + page file) and 3.7 GiB of it is left, but this run plans 2.0 GiB of host allocations and keeps 4.0 GiB free: it can end at the limit with no message (exit code 0xC0000409, #1607) while RAM is still free. Raise the page file (System managed, or RAM-sized) or lower --conversation-cache-mib / --max-context / --prefill
  strata: prefill plan: chunk 8192, ring 96 kept: file set 10.4 GiB does not fit the page cache 0.0 GiB -> about 44.6 GB read from the drive per 32768-token prompt (4 chunks); no option does better (the set cannot fit: fewest passes win)
  strata: prefill plan inputs: 71.7 GiB experts, 34.8 GiB in the GPU caches, 34.3 GiB RAM copy (0 lend slots kept), file tier 2.6 GiB; 0.0 GiB available for the page cache (the file tier reads unbuffered: none); PCIe 3.2 GB/s; drive 1.50 GB/s (assumed); a 32768-token prompt
  strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
  strata serve: the prompt path borrows 1276 CUDA0 cache slots (3.73 GiB)
  strata serve:   CUDA1 prompt path borrows 1270 of its 7931 slots (3.73 GiB)
  strata serve: lend sizing: the prompt path borrows 2546 slots (ring 96 slots at chunk 8192), 0 of them keep their experts in RAM too
  strata serve: WARNING: 2546 of the 2546 borrowed slots have no RAM copy: their experts are streamed from the model file during every chunk): the default settings stay
       the engine said: strata serve: WARNING: 2546 of the 2546 borrowed slots have no RAM copy: their experts are streamed from the model file during every chunk
  [ok] start script: run-unsloth-ud-q4_k_xl.bat

All set.
  API (OpenAI):     http://127.0.0.1:8080/v1   (any API key; model name: anything)
  API (Anthropic):  http://127.0.0.1:8080/v1/messages
  Next time:        just run START-HERE.bat (or run-unsloth-ud-q4_k_xl.bat) - it starts right away
  Images:           send them in the chat page, in chat.py (/image <path>) or over the API
  Several requests at once (opt-in): --parallel 4 decodes up to 4 together instead of one after the other (each takes ~0.6 GB of VRAM from the expert cache; docs/BATCHING.md).
  Tuning:           FAILED (the reason is above): the default settings stay - START-HERE.bat --calibrate tries again
Run a quick benchmark now? It measures this install for about 10 minutes and writes a report you can share (later: BENCH.bat) [n]:
stopped.
Terminate batch job (Y/N)?
^CP

Mehr auf der Site

Links zu Install, Modellen, Releases.