Issues / #2856

#2856 Significant regression after 0.1.45 on 32 GB RAM + 6 GB VRAM system (previously working config doesn't work anymore)

open · @Dampfinchen · 0 Kommentare · Auf GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows

Beschreibung

I have previously been using this configuration:

`{
 "exe": "D:\\KI\\Strata\\engine\\strata.exe",
 "args": [
  "--pack",
  "D:\\KI\\Strata-data\\packs\\iq3_xxs",
  "--native",
  "D:\\KI\\Strata-data\\models\\IQ3_XXS\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "--ple-gguf",
  "D:\\KI\\Strata-data\\models\\IQ3_XXS\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf",
  "--expert-profile",
  "D:\\KI\\Strata\\data\\expert-profile.bin",
  "--expert-cache",
  "auto",
  "--prefill",
  "auto",
  "--spec",
  "4",
  "--max-context",
  "81920",
  "--mmap-experts",
  "--kv",
  "k8v4",
  "--vram-reserve-mib",
  "150",
  "--spec-min-p",
  "0.5",
  "--adapt-every",
  "1",
  "--adapt-swaps",
  "80",
  "--adapt-decay",
  "0.97"
 ],
 "cwd": "D:\\KI\\Strata",
 "tokenizer": "D:\\KI\\Strata-data\\packs\\iq3_xxs\\tokenizer",
 "model_name": "qwen3.8-flash-next-iq3_xxs",
 "log": "D:\\KI\\Strata\\strata-iq3_xxs.log",
 "lib_dirs": [
  "D:\\KI\\Strata\\.venv\\Lib\\site-packages\\nvidia\\cu13\\bin\\x86_64"
 ],
 "port": 8080,
 "gpu": 0,
 "gpus_asked": true,
 "draft_vocab": "en"
}`

With an earlier version this worked while having 41 experts on GPU, around 13 token/s decode, so quite usable if it not were for the horrendously slow prompt processing which was around 50 token/s. But it worked. Now it only caches 14 experts and I'm getting this error as soon as I send the prompt:

`[strata] the engine stopped unexpectedly (exit code 3221226505). The engine exited (code 3221226505). Its last log line: strata serve: WARNING: only 0 MiB of VRAM free at the start of this request (below 1024 MiB): when the card is this full the driver can page GPU memory to system RAM, decode falls to a few tok/s and a window may never finish (#2065). Add --vram-reserve-mib 1216 to the config's args, or lower --max-context / --batch - if that does not explain it, please report it at github.com/Niko1221/Strata/issues with the log. The next request starts the engine again. Its log: D:\KI\Strata\strata-iq3_xxs.log`

Reducing --vram-reserve-mib to 120 it doesn't even boot anymore (it did previously)

` strata serve: Windows commit: 23.1 GiB left of a 37.1 GiB limit (RAM + page file)
  strata: the CPU prefill share is not armed by default: the experts it would read are 39.9 GiB of file pages and only 23.2 GiB of RAM is free to keep them. STRATA_PREFILL_CPU_SHARE=auto turns it on anyway
  strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
  strata serve: prefill: device buffers for a chunk of 1024 tokens do not fit (0 of 6143 MiB free at the failure; 470 MiB taken, the failing buffer wanted 10 MiB): trying a 512-token chunk
  strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
  strata serve: verify: the device arena (69 MiB) does not fit`

Setting the env variable STRATA_PREFILL_CPU_SHARE=auto doesn't help and only gives me this error:

` strata serve: Windows commit: 23.0 GiB left of a 37.1 GiB limit (RAM + page file)
  strata: WARNING STRATA_PREFILL_CPU_SHARE=auto is set and kept, but the experts it would read are 39.9 GiB of file pages and only 23.1 GiB of RAM is free to keep them: the share can slow prompts or fail on a machine this short of memory (STRATA_PREFILL_CPU_SHARE=0 turns it off)
  strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
  strata serve: prefill: device buffers for a chunk of 1024 tokens do not fit (0 of 6143 MiB free at the failure; 470 MiB taken, the failing buffer wanted 10 MiB): trying a 512-token chunk
  strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
  strata serve: verify: the device arena (69 MiB) does not fit`

Setting STRATA_PREFILL_CPU_SHARE=0 doesn't help either. 

[strata-iq3_xxs.log](https://github.com/user-attachments/files/33591076/strata-iq3_xxs.log)



Mehr auf der Site

Links zu Install, Modellen, Releases.