#1878 [BUG] I think that putting experts on each card might be broken?

Descripción

I was trying to use Strata with Hermes Agent today.  Last night it was working fine with v0.1.4.1, but after the update and setting up, I was getting this:

00000000000

Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)

=== Step 1: checking your PC ===
  Your NVIDIA GPUs:
    GPU 0: NVIDIA GeForce RTX 3060, 12 GB VRAM - can be used
    GPU 1: NVIDIA GeForce RTX 4090, 22 GB VRAM - can be used
    GPU 2: NVIDIA GeForce RTX 3060, 12 GB VRAM - can be used

  Strata can run the model on one GPU, or share it across these: then each card holds the
  experts of its own layers, so together they hold about twice as many, and prompts are read about 20%
  faster (details: docs/MULTI_GPU.md). A much slower extra card can also make it slower.
  1) GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) + GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) together   (recommended)
  2) GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) + GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) + GPU 2 (NVIDIA GeForce RTX 3060, 12 GB) together
  3) GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) only
  4) GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) only
  5) GPU 2 (NVIDIA GeForce RTX 3060, 12 GB) only
Which GPUs? [1]: 2
  [ok] GPUs: GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) + GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) + GPU 2 (NVIDIA GeForce RTX 3060, 12 GB) together (the model's layers are split across them)
  [ok] GPU: NVIDIA GeForce RTX 4090, 22.5 GB VRAM, compute capability 8.9, driver 617.42
  [ok] RAM: 127 GB
  [ok] CPU: AMD Ryzen Threadripper PRO 7965WX 24-Cores (AVX-512)
  [ok] PCIe: 4.0 x16

=== Step 2: your choices ===
  1) Qwen3.8-Flash-Next   Qwen; GSQ-RCO quants by ISTA-DASLab - the original model
  2) Swift 1.5            UkisAI's fine-tune of Qwen3.8-Flash-Next - thinks much shorter (-63% thinking tokens, 1.8x sooner answers by its authors' numbers)
  3) Qwen3.8-Flash-Next Coder ISTA-DASLab's coding version - half the experts (code, tools, images kept): needs ~32 GB of RAM, faster; weaker outside coding
  4) Qwen3.8-Flash-Next (Unsloth) Unsloth's ~4-bit quantizations - UD-IQ4_XS: a 94 GB download; with less than ~80 GB of RAM part of its experts are read from the SSD (UD-Q4_K_XL, 111 GB: experimental)
Which model? [1]: 4
  [ok] model: Qwen3.8-Flash-Next (Unsloth)

  1) UD-IQ4_XS ~4-bit i-quant (Unsloth Dynamic), between IQ3_S and UD-Q4_K_XL in quality; on a PC with less than ~80 GB of RAM part of its experts are read from the SSD; download 94 GB, keeps ~55 GB of its 60 GB of experts in RAM
  2) UD-Q4_K_XL 4-bit (Unsloth Dynamic), EXPERIMENTAL: the best quality, but most experts come from the SSD on a 64 GB PC (7-8.5 tokens/s measured); download 111 GB, keeps ~71 GB of its 77 GB of experts in RAM
Which size? [1]: 2
  [!]  UD-Q4_K_XL is EXPERIMENTAL (docs/UNSLOTH_Q4.md): most of its experts are read from the SSD while it answers, so it is several times slower than the 2-3-bit models; quality checked against llama.cpp

  UD-Q4_K_XL runs with a RAM budget of its experts, on one GPU or on GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) + GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) + GPU 2 (NVIDIA GeForce RTX 3060, 12 GB) together:
  1) GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) only   (recommended: the tested setup)
  2) GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) + GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) + GPU 2 (NVIDIA GeForce RTX 3060, 12 GB) together: each card caches the experts of its own layers (#642)
UD-Q4_K_XL: which GPUs? [1]: 2
  [ok] UD-Q4_K_XL on GPU 1 (NVIDIA GeForce RTX 4090, 22 GB) + GPU 0 (NVIDIA GeForce RTX 3060, 12 GB) + GPU 2 (NVIDIA GeForce RTX 3060, 12 GB), with its RAM budget (each card caches its own layers' experts, the RAM the hottest of the rest; #642)
  [ok] RAM budget: 71 GiB of UD-Q4_K_XL's experts in RAM, the rest read from the model files on the SSD
  [ok] size: UD-Q4_K_XL

  Context length = how much text the model can see at once (your chat, files, tool output).
  Longer needs more VRAM for it, so fewer experts fit on the GPU:
  1) 8K tokens   (recommended for your GPU)
  2) 32K tokens
  3) 64K tokens
  4) 128K tokens
  5) 200K tokens
  6) 256K tokens
  7) 384K tokens   (experimental: setup adds rope scaling)
  8) 512K tokens   (experimental: setup adds rope scaling)
Context? [1]: 6
  [ok] context: 262144 tokens

  KV cache precision (the model's memory of the conversation):
  1) 8-bit   (recommended: what every published number was measured with)
  2) 4-bit   half the memory (about 4% faster at 128K), but measurably less precise on long
             documents; long-context lookups (needle tests) still pass
KV cache? [1]: 1
  [ok] KV cache: 8-bit

  Images: the model can also read pictures (screenshots, photos, scanned pages). This adds a 0.9 GB
  download and keeps ~1.4 GB of VRAM free for the image encoder, so text is a few % slower.
Do you want images? [n]: y
  [ok] images: on
  [!]  images with UD-Q4_K_XL are untested: users report them working, but we have not run this file with images (#967); tell us if the answers look wrong

=== Step 3: Python packages ===
  [ok] numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil already installed

=== Step 4: the Strata engine ===
  [ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
  Updating the ready-made engine (0.1.41 -> 0.1.42 or newer) ...
  Downloading the ready-made Strata engine ...
  Strata engine:   0.13 / 0.14 GB (98%)
  [ok] Strata engine downloaded
  [ok] strata-windows-x64.zip: SHA-256 checksum verified against GitHub's (bbf2fca32f98...)
  [ok] the engine it replaced (0.1.41) is kept in C:\Users\Janus\Desktop\ARTIFICIAL\Strata-main\engine\.previous; setup.py --rollback-engine puts it back
  [ok] ready-made engine 0.1.42 for sm_75, sm_86, sm_89, sm_120 (CUDA 13.0)
  [ok] NVIDIA CUDA libraries (cuBLAS, CUDA runtime; ~0.4 GB) already installed
  [ok] engine: C:\Users\Janus\Desktop\ARTIFICIAL\Strata-main\engine\strata.exe

=== Step 5: downloading Qwen3.8-Flash-Next (Unsloth) UD-Q4_K_XL ===
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf already downloaded
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf already downloaded
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf already downloaded
  [ok] Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf already downloaded
  [ok] model files present
  [ok] vision encoder already downloaded
  [ok] vision encoder: C:\Users\Janus\Desktop\ARTIFICIAL\Strata-data\models\mmproj-Qwen3.8-Flash-Next-BF16.gguf

=== Step 6: preparing the model for Strata ===
  [ok] model prepared: C:\Users\Janus\Desktop\ARTIFICIAL\Strata-data\packs\unsloth-ud-q4_k_xl
  [ok] MTP draft layer: C:\Users\Janus\Desktop\ARTIFICIAL\Strata-data\mtp\rt

=== Step 7: writing the start script ===
  [ok] KV streaming on: the context's KV cache lives in RAM (3.6 GB), more experts fit in VRAM
  [ok] RAM budget: 71 GiB (less the KV cache's RAM)
  [ok] layer split across GPUs [1, 0, 2] (auto)
  tip: with 127 GB of RAM, --prefill auto:32768 in the config's args read prompts 21-35% faster in community benchmarks (#433 #440 #834); not set, nothing changes
  tip: for several agents or clients at once, --conversation-cache-mib 8192 in the config's args keeps each one's conversation; without it they were measured re-reading ~90% of their prompts (#882 #440; docs/DETAILS.md, Multiple conversations)
  the earlier run config is kept as strata-unsloth-ud-q4_k_xl.json.bak; engine options it had that this one has not (setup chooses those): --pool-workers --adapt-every --adapt-swaps --adapt-decay
Tune Strata for this PC now? It measures a few engine settings (about 15-30 minutes, longer on a slow card; the PC is busy meanwhile; later: START-HERE --calibrate) [y]: y

  Tuning Strata for this PC: the output speed is measured with a few engine settings (the PCIe share, the
  draft depth, the CPU threads, the expert cache). It takes roughly 15-30 minutes (3 rounds of each value); the PC is busy meanwhile.
  Loading the model for the measurements ...
[strata] layer split: card order 0,2,1 (the faster card last; "gpu_order": "as_given" keeps 1,0,2, #1352)
[strata] layer split: card order 0,2,1 (the faster card last; "gpu_order": "as_given" keeps 1,0,2, #1352)
[strata] starting the engine: reading the model's weights ...
[strata] still starting (20 s) - please wait ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
         YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
         Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (1536 experts, 4.83 GiB of VRAM) ...
[strata] still starting (46 s) - please wait ...
[strata] still starting (66 s) - please wait ...
[strata] almost ready ...
[strata] still starting (101 s) - please wait ...
[strata] still starting (121 s) - please wait ...
[strata] still starting (142 s) - please wait ...
[strata] still starting (162 s) - please wait ...
[strata] still starting (182 s) - please wait ...
[strata] still starting (203 s) - please wait ...
    round 1/3 PCIe share: 0 29.1  0.2 30.3  0.35 27.8  0.55 32.5  0.75 29.2  0.9 27.8  1 26.2
    round 2/3 PCIe share: 0 22.9  0.2 26.5  0.35 26.4  0.55 31.4  0.75 30.5  0.9 28.3  1 30.7
    round 3/3 PCIe share: 0 23.1  0.2 23.6  0.35 26.3  0.55 26.7  0.75 26.5  0.9 28.7  1 29.1
    PCIe share, median of 3: 0: 23.1 (22.9-29.1), 0.2: 26.5 (23.6-30.3), 0.35: 26.4 (26.3-27.8), 0.55: 31.4 (26.7-32.5), 0.75: 29.2 (26.5-30.5), 0.9: 28.3 (27.8-28.7), 1: 29.1 (26.2-30.7) -> best 0.55
    round 1/3 draft floor: 0.3 23.3  0.5 21.5  0.7 29.5
    round 2/3 draft floor: 0.3 30.5  0.5 23.8  0.7 25.4
    round 3/3 draft floor: 0.3 30.3  0.5 24.2  0.7 22.7
    draft floor, median of 3: 0.3: 30.3 (23.3-30.5), 0.5: 23.8 (21.5-24.2), 0.7: 25.4 (22.7-29.5) -> best 0.3
  Measuring with 23 CPU workers (restarts the engine) ...
[strata] layer split: card order 0,2,1 (the faster card last; "gpu_order": "as_given" keeps 1,0,2, #1352)
[strata] layer split: card order 0,2,1 (the faster card last; "gpu_order": "as_given" keeps 1,0,2, #1352)
[strata] starting the engine: reading the model's weights ...
[strata] still starting (20 s) - please wait ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
         YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
         Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (1536 experts, 4.83 GiB of VRAM) ...
[strata] still starting (45 s) - please wait ...
[strata] still starting (66 s) - please wait ...
[strata] still starting (86 s) - please wait ...
[strata] still starting (107 s) - please wait ...
[strata] almost ready ...
[strata] still starting (130 s) - please wait ...
[strata] still starting (151 s) - please wait ...
[strata] still starting (171 s) - please wait ...
[strata] still starting (191 s) - please wait ...
[strata] still starting (212 s) - please wait ...
[strata] still starting (232 s) - please wait ...
    23 workers: 35.8 tok/s
  Measuring with 15 CPU workers (restarts the engine) ...
[strata] layer split: card order 0,2,1 (the faster card last; "gpu_order": "as_given" keeps 1,0,2, #1352)
[strata] layer split: card order 0,2,1 (the faster card last; "gpu_order": "as_given" keeps 1,0,2, #1352)
[strata] starting the engine: reading the model's weights ...
[strata] loading the most-used experts into RAM (up to 71 GiB; the rest are read from the model files as needed) and locking part of them for the GPU.
         YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
         Please wait and don't close this window; the browser opens when it is ready.
[strata] filling the GPU's expert cache (1536 experts, 4.83 GiB of VRAM) ...
[strata] still starting (45 s) - please wait ...
[strata] still starting (65 s) - please wait ...
[strata] sti