#6617 [Engine] Inkling-2-Mini SQ-20 on RTX 3080 12 GB: wrong tokens with experts on the host and pf_buf=0MB (--cpu is correct)
open · @balaena01 · 0 comentários · No GitHub
BenchmarksSetup & installNVIDIA / CUDAModels & quantsWindowsLinux
Descrição
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Desktop app (Windows, Linux or Mac) ### What happened The official, unmodified `Inkling-2-Mini-SQ-20.strata` loads without an error on an RTX 3080 12 GB, but the answers are wrong from the first token: unrelated tokens in many languages, then loops, at about 3 tok/s in the desktop app. The same pack with `--cpu` gives a sensible answer, so the pack is fine and the single-GPU path with experts on the host is not. `--selfcheck --verify-hash` passes on the same file. Load line (every run): ``` [InkCUDA] Loaded …\Inkling-2-Mini-SQ-20.strata (SQ-20): trunk=3.67 GB (2.25s), experts=1282 VRAM / 1790 host (9.74s, hyb=1, promo=0, pf_buf=0MB), sm120_tc=0, screen=0 [InkCUDA] Auto-sizing max_ctx 32768 -> 28928 for parallel=1 (1.05 GiB free VRAM) ``` Expected: the same tokens as `--cpu`, or a refusal to load if the staging buffer cannot be allocated. ### Smallest steps or command that reproduce it Run directly on the engine the app installed (`…\components\shapes\inkling2\engine\0.1.65-windows-x86_64-cuda-cuda13\strata-inkling.exe`), drafter off, fixed prompt ids (system "§", user "¨", then the model turn): ``` strata-inkling.exe --native Inkling-2-Mini-SQ-20.strata --draft off --gen 24 --prompt-ids 200002,200004,100,200010,200000,200004,101,200010,200001,200004 ``` Output (GPU, experts split 1282 VRAM / 1790 host, `pf_buf=0MB`): ``` 48876,220,19,220,68205,33,149897,19,33,220,19,12,1261,220,25208,220,48876,220,19,220,68205,33,149897,19 ``` which decodes to `Gemma 4 26B A4B 4-bit 的 Gemma 4 26B A4…`. Same command with `--cpu`: ``` 371,151090,149941,150678,156090,149882,149895,153436,149918,152068,150130,149916,189978,14,3932,12,107440,151930,13,179027,149941,150592,149991,150425 ``` which decodes to `The message you sent appears to be empty or contains only an invisible/non-printing character. Could you let me know`. Two controls on the same machine: - **One-token prompt** (`--prompt-ids 200002`, `--gen 16`): GPU gives `12,3371,68014,54081,68014,36,855,131117,68014,2519,150200,196986,90948,196973,3371,13`, `--cpu` gives `12,3849,12,33526,257,68014,16,198,3,38,124870,3,198,16,13,15`. The first token agrees and both continuations are coherent text; nothing like the garbage above. - **SQ-10, all experts in VRAM** (`experts=3072 VRAM / 0 host (hyb=0, promo=0, pf_buf=0MB)`), same 10-id prompt: `371,150409,150008,150678,149878,150971,151930,25,196980,101,1,13,150078,149884,149878,220,35068,151930,196976,64,220,35068,153151,11` = `The user has sent a single character: "¨". This is a Hebrew character (a Hebrew letter,` — sensible. **Suspected cause — unconfirmed static analysis of the v0.2.19 tag (0.1.65); I did not patch or rebuild to confirm it:** - In the multi-token path of `launch_ink_moe_hybrid` (`engine/src/inkling2/ink_hybrid.cuh`, around lines 984–1054) the host experts are staged into `d_pf_stage` and computed on the GPU. When the staging buffer was not allocated (`pf_stage_bytes == 0`, around line 999) the list of splits is empty, so the host experts are not computed, and there is no fallback. - `k_ink_moe_reduce8` (`ink_moe_cuda.cuh`, around lines 324–337) still sums all 8 slots of `ws.y_pair`, and I could not find where `y_pair` is zeroed, so the slots of the skipped experts would keep values from an earlier layer. That would fit output made of real tokens rather than noise. - The 256 MiB for staging is budgeted before the experts are placed (`ink_runner.cuh`, around lines 221–224), but free VRAM is measured again after loading (lines 266–270) and the buffer is not allocated when what is left is 48 MiB or less (`ink_hybrid.cuh`, around lines 807–818). Here that leaves `pf_buf=0MB` on every run. - Single-token steps use the doorbell path (`ink_hybrid.cuh`, around lines 956–967), which would explain why the one-token prompt above stays coherent while a 10-token prompt is wrong from the first generated token. What would verify it: allocate the staging buffer before placing the experts (or fail the load when it cannot be allocated) and zero `y_pair` at the start of the multi-token path, then compare the token ids with `--cpu` for the command above. Possibly related: #6569 (host-expert computation on a 12 GB card), #6611 (expert placement on Windows). ### Reproduction status Reproduced locally (The wrong tokens are reproduced on every run. The cause above is from reading the source only.) ### Strata version Strata 0.2.19 · engine strata-inkling 0.1.65 (windows-x86_64-cuda-cuda13), installed with Strata-0.2.19-windows-x64-setup.exe ### Graphics card, memory and system RTX 3080 12 GB (sm_86, 70 SMs) · 32 GB RAM · Windows (driver version not recorded) ### Model, size and settings Inkling-2-Mini SQ-20 (and SQ-10 as a control) · app default context (131072 requested, auto-sized to about 28k) · pictures on in the app; the commands above pass no `--vision` ### Logs ``` [selfcheck] Verifying SHA-256 of Sections 0..5... [InkCUDA] Loaded …\Inkling-2-Mini-SQ-20.strata (SQ-20): trunk=3.67 GB (2.54s), experts=1282 VRAM / 1790 host (11.41s, hyb=1, promo=0, pf_buf=0MB), sm120_tc=0, screen=0 [InkCUDA] Auto-sizing max_ctx 32768 -> 28736 for parallel=1 (1.05 GiB free VRAM) [selfcheck] PASS backend=cuda (NVIDIA GeForce RTX 3080 (sm_86, 70 SMs, 12.0 GiB VRAM, sm120_tc=0)) ``` Source reading was done with AI assistance; the commands and outputs above are from my own machine.
No site
Links install, modelos, releases.