Temas / Multi-GPU
Multi-GPU
Auto topic: Multi-GPU
Issues
- #6442 [Engine] Inkling-2-Mini support (looped MoE from Thinking Machines)
- #6537 [Engine] Keep evicted parked conversations on disk, also with --layer-split and --batch
- #5939 [Engine] Windows CUDA12 0.1.59: Tesla V100 GPU is lost during single- and dual-GPU inference
- #6530 [Engine] Hy4-Flash SQ-64 (strata-hy4 0.1.64): long reasoning falls into an endless repetition loop (2 of 2 attempts)
- #6531 [Benchmark] gfx1100 (RX 7900 XTX, IQ3_S): existing switches give 1.9–3.7x prompt and 1.2x decode speedup
- #6523 [Engine] Windows NUMA in Strata: Why Our 40-Core Server Initially Used Only 20 CPU Workers
- #6526 [Engine] gfx1200 (2x RX 9060 XT, ROCm 10.1): peer-GPU fused WMMA prefill, RDNA4 MMVQ load_int_b2/Q6KTraits, and hipBLASLt fallback notes
- #6525 [Engine] Drop in MTP head replacement trained against quantized model output: +3.7 / +4.4 % decode at --spec 3
- #5614 [Engine] RX 9070 XT gfx1201: 0.1.58 verify unspecified launch failure triggers MES REMOVE_QUEUE GPU reset
- #6511 [Benchmark] 2x Tesla P40 (sm_61) split: 0.2.17 gives +9% decode, but 220K prefill is still ~5% under the 0.2.13.1 MMQ path (same-day A/B)
- #6431 [Engine] 0.1.63 STRATA_L2_PF=1 default regresses NVFP4 batch decode by 8-12% on 2x RTX 5090 with PCIe-streamed experts
- #6232 [Engine] 384K compaction re-reads 264K tokens (456 s): content-addressed, position-independent segment cache for near-zero-reread compaction
- #6509 [Engine] Engine 0.1.55+ decodes 35–55% slower than 0.1.52 on RTX 3080 Ti + RTX 4070 (Windows): decode expert-cache hit falls from ~92% to 55–70% although the cache gets ~40% more slots
- #6507 [Engine] q36: one early byte change costs the whole prompt — all-or-nothing match_len has no GDN state to step back to (0 reused + 38026 read, 108 s)
- #6327 [Engine] Compaction re-reads 331K-342K tokens five times (19.5-20.1 min each): the cache budget is the blocker (PR #4338)
- #6480 [Engine] qwen36 baked chat template breaks prompt reuse
- #6472 [Benchmark] RTX 4090 24 GB VRAM headroom for long-context KV and IQ4 experts
- #6404 [Engine] gfx1030 (RX 6800 XT, Windows): 0.1.63 reads prompts 2x faster than 0.1.60 but decodes 11 % slower — same config, A/B on one host
- #5990 [Engine] V100 (sm_70) on Linux: 13× Xid 79 + PCIe CmpltTO across engines 0.1.35–0.1.59.1 — not version-specific
- #6118 [Engine] Cold 50K prefill regression: 0.1.52 → 0.1.59 → 0.1.60 on RTX 3060 12GB (−16.5%)
- #6426 [Engine] --layer-split auto with DFlash2 on two cards: the noisy PCIe ratio around 4.0 picks a placement with no expert room, and the drafter is dropped
- #6271 [Other] Unable to build docker image of Strata
- #6423 [Engine] 0.2.17 sm_120 defaults regress prefill and cache-reuse TTFT on a mixed-arch layer split with the 12.0 card first
- #6421 [Benchmark] Ryzen 7 5700X, 64GB DDR4-2400, 2x RX 7800 XT 16GB (Ubuntu/ROCm) -- Qwen3.8-Flash-Next UD-IQ4_XS (256K, pinned 41.4 GiB, --spec 4)
- #6145 [Engine] Deterministic #879 with --ple-io direct on UD-Q6_K_XL (mmap passes; expert blob bytes certified clean; boot-state dependent)
- #6160 [Engine] STRATA_PIPED_MTP is turned off on Linux CUDA by the engine's own parallel defaults (STRATA_BATCH_SELF_COMMIT=1)
- #6417 [Engine] 0.1.63: after one picture through the elastic image encoder, the expert cache can grow back a segment short and --prefill auto:32768 sticks at 16384 (60K cold prompt -37 %)
- #6413 [Benchmark] v0.2.17 on 2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K: same speed as v0.2.16, `--pipeline-windows 2` no longer produces `!!!!` (12 of 12)
- #6405 [Engine] 0.2.17: on a 3-card layer split the expert cache shrinks from 5,940 to 3,551 slots (10.40 → 6.19 GiB) at the same VRAM footprint
- #6400 [Engine] + [Docs] SYCL setup & LXC/multi-GPU flow is a complete mess (paths, build dir, device selection)
- #6156 [Engine] Qwen3.6-35B-A3B on 2x RTX 3060: qwen35moe ignores multi-GPU flags — patch adds a card-1 expert tier, 37.6 → 74.8 tok/s (+97%)
- #6397 [Engine] Engine cannot use more than 8 videocards
- #6052 [Engine] Equal prompt chunks skip periodic conversation checkpoints (--prompt-cache-every)
- #6265 [Engine] layer-split auto: the #2644 clamp picks a slower split on asymmetric same-VRAM pairs (measured +10% hand-pinned)
- #6171 [Engine] 0.2.15 window warm-up with STRATA_ROUTE_PRIOR: the first request after a start often ends in the middle of a number
- #6111 [Engine] STRATA_OWN_CPU_QUANT=ON (0.2.15 x86_64 default) is ~4% slower than ggml-cpu on AVX2 CPU expert pool decode (IQ3_XXS)
- #6110 [Engine] explicit --layer-split reloads CUDA0 dense weights a second time at startup (0 MiB freed)
- #5950 [Engine] STRATA_COALESCE_PREFILL never co-reads on a piped layer split or for a burst reaching an idle server
- #5895 [Engine] Long agent sessions on 64 GB RAM: a 260k conversation can't be parked, so every side request forces a full reread (follow-up to #57 / #190)
- #5822 [Engine] Flash-Next 0.1.58: native QSA indexer (batch) rejects arguments and exits on single-GPU CUDA
- #5718 [Engine] gfx1200 (RX 9060, Linux): "verify: timed out" (#267) during decode, not tied to spec decoding, RAM budget or pool workers
- #5654 [Engine] Dual RTX 5060 Ti over OCuLink: Xid 79 during tune auto (0.1.58, Linux driver 615.71.09 open)
- #5612 [Engine] conversation cache: on a layer split, a sub-agent that matches only a shared head takes the whole parked conversation and drops its checkpoints
- #6389 [Benchmark] 5 x Quadro RTX 8000 (sm_75), DeepSeek-V4.1-Flash IST 2.25 bpw: decode 13.5 to 29.3 tok/s by expert placement and --threads 1
- #6236 [Other] RTX 4090 / Qwen3.8: 131K Context, Performance & CUDA Stability — Field Report
- #6218 [Engine] No character-before-the-tag rule settles </think>: four shapes, three tickets, two opposite fixes
- #5981 [TUI] No individual PCIe current lane width for multiple GPUs
- #5747 [Engine] SYCL layer split: serve plans DFlash2 per the CUDA/HIP rule, the fork refuses (generate.cpp:4425) but exits 139 SIGSEGV - fallback-none crash-loops the service
- #6388 [Other] Docker: a key taken out of STRATA_SET stays in the saved preset and still reaches the engine
- #6381 [Benchmark] 6x Tesla V100 16GB (sm_70): 0.1.63 with exact routing - prefill parity kept, decode +7% over the rank-7-skip 0.1.62, and the single-stream prefill gain confirmed on this rig
- #6380 [Engine] Conversation cache: 0 reused on a single GPU since 0.2.11 (host layer); two-GPU layer split unaffected; engine version independent
- #6030 [Engine] CLI path: a native (IQ) pack's residual is never seeded, so output is pinned to token 0 (verified on sm_75 and sm_89; standard GGUF packs are unaffected)
- #6374 [Engine] Scope: reuse normalized BF16 hyper-connection inputs on NVIDIA GB10
- #6361 [Benchmark] Unequal NVIDIA pair, both card orders, `--layer-split auto`, and the helper mode: RTX 3060 12 GB + RTX 5060 Ti 16 GB, 0.2.16, NVFP4-GPTQ
- #6359 [Benchmark] Tesla V100 (sm_70): opt-in decode kernels on 0.2.16, decode +13..24% on 2x V100-SXM2-32GB
- #6355 [Engine] Layer-split serves cannot reach either disk cache: the SSD spill tier and session files are single-serve only
- #6354 [Benchmark] RTX 3060 12 GB on an AVX-only Xeon E5-2630 v2 (no AVX2/FMA): 0.2.16 (engine 0.1.62) against the 0.2.0 report, IQ2_XS and NVFP4
- #5549 [Benchmark] Single vs dual rx9070 (hip): engine 0.1.58, ud-iq4_xs-v31
- #6351 [GUI] make the default reading size 14 px and give every text role a fixed size table, so nobody has to open Settings
- #6349 [Engine] Initial prefill batch length changes layer-3 KV and greedy termination with zero checkpoint reuse
- #6141 [Engine] Scope proposal for NVLink expert uploads on asymmetric PCIe links
- #6322 [Engine] v0.2.16 source build on gfx1201 segfaults before any output with system ROCm 7.16; STRATA_HIP_QUEUE_PRIME=0 and the official rx9070 package are unaffected
- #6279 [Engine] RTX 4090 (Windows, CUDA 13): decode about 10-20% slower on 0.1.62 vs 0.1.60 with UD-IQ4_XS-v31 (prompt reading unchanged)
- #6308 [Engine] RTX 4090 + RTX 3090 layer split: 0.1.62 reads prompts 6-12% slower than 0.1.59 (prompt experts moved from MMQ to the fused int8 kernels)
- #6298 [Benchmark] 24 GB vs. 48 GB: DeepSeek V4.1 Flash with Strata (RTX 3090 Ti vs. Quadro RTX 8000)
- #6295 [Benchmark] v0.2.16 on 2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K: same speed as v0.2.13, `!!!!` with --pipeline-windows 2 back to 0 of 12
- #6296 [Other] 0.2.16 layout: Intel (SYCL) setup cannot start the server ("cannot open .../engine/run-*.sh"), and `strata bench run` from a git clone is always nonconforming
- #6290 [Engine] gfx1100 + UD-IQ4_XS-v31: ask for a Q5_0-down route in the fused prompt path (our int8 extension measures +55% prefill at 114K)
- #6038 [Engine] gfx1030 (2x Radeon PRO W6800 64GB, Linux HIP 0.1.59): DeepSeek-V4.1-Flash @ 256k context benchmarks, >15k multi-chunk prefill deadlock (#4211), and DS4_COPY_PIN kfd_ioctl stall
- #5922 [Engine] DeepSeek V4.1 on a 12 GB card: --dspark on gives the drafter every VRAM slot and the prompt path falls to windows of 16; a cap on the drafter's VRAM experts makes it a net gain
- #5570 [Engine] DeepSeek-V4.1 has no --no-prefill-borrow: on a 16 GB card the prompt loan costs half the expert cache
- #6242 [Benchmark] 6x Tesla V100 16GB (sm_70): 0.1.62 parity on concurrent prefill - the +16% single-prompt gain does not transfer to 4x26K concurrency
- #5564 [Engine] Qwen3.8-Flash-Next IQ3_XXS 0.1.58: prefill stuck at ~111K tokens — cancel leaves the engine busy-waiting, GPU 0%, one request slot — reproduced on 2× RTX 2080 Ti sm_75
- #6234 [Other] RTX 4090 + Qwen: 131K Context, Performance, CUDA Stability — Community Field Report
- #5560 [Engine] STRATA_HC_REQ8=1 with STRATA_HC_PERSIST=1 (HIP): the persistent hc read takes the int8 codes as bf16
Pull requests
- #4310 [Engine] session files: format v2 with per-stage save/restore under --layer-split
- #6520 [Engine] Inkling-2-Mini from the official NVFP4/MXFP8 checkpoint, expert offload and CPU experts for 8-16 GB cards
- #6023 [Engine] peer_stream_ring: fix >= 2K peer_i8 ring wraparound + V100 docs
- #6210 [Engine] Opt-in P2P q8_1 decode push, fused P2P down-projection, and GPU peer sync flag
- #6529 [Engine] Volta sm_70 decode optimizations, zero-VRAM in-place Q8_0, NVLink split head, and volta-forever detection
- #6548 [Engine] Add opt-in streamed-KV QSA slot-group batching, pre75 multi-slot attention, and batch SIDE_MIXER
- #6547 [Engine] Add opt-in multi-slot batched MTP drafting (STRATA_BATCH_MTP_FUSED)
- #6546 [Engine] Opt-in resize_for_overwrite and parallel host memcpy for kv_mode != 0 park/restore
- #6544 [Engine] Fix O(N^2) scan in getline_fd and avoid full-line copy in BGEN/parse_i64_list
- #4551 [Engine] session-file v3: incremental append saves under --session-save-append
- #3773 [Engine] q36: MTP drafts for sampled requests behind STRATA_Q36_MTP_SAMPLED (stacked on #3678)
- #6532 [Engine] prefill: opt-in STRATA_I8_DEFER - unpicked streamed entries no longer cut the int8 expert groups into fragments (gfx1030 4K prompt +3.6%)
- #5436 [Engine] opt-in STRATA_ARENA_NUMA: place the expert arena per NUMA node on two-socket Linux
- #6506 [Engine] add Inkling 2 model family and dual-GPU looped MoE engine support
- #6399 [Engine] Prefetch routed expert weight rows into L1 on GB10
- #4928 [Engine] long --lookup-max-t windows over a layer split (opt-in, single conversation)
- #4376 [Engine] dsv41: the WMMA expert kernels compile on pre-Volta cards
- #6197 [Engine] int8 expert kernels: IQ4_XS as a down format
- #6501 [Engine] Give each layer-split stage its own two-pass check buffers
- #6498 [Benchmark] RTX 3060 12GB / Qwen3.8 Flash-Next Q2_0 / standard v1 draft.1 + 260K xhigh
- #6419 [Engine] autonomous hardware-synchronized dual-GPU decode window (P2P zero-wait) [Draft speed Lab 230 tok/s+ RFC]
- #6481 [Engine] keep the baked qwen think block when preserve_thinking is unset
- #2965 setup: a separate opt-in layers 8-33 projection vector for the original Qwen3.8-Flash-Next (any quantization)
- #6477 [Benchmark] Arc Pro B60 / Qwen3.8-Flash-Next / IQ3_S / strata-qwen38-thinking-v1-draft.1
- #3957 [Engine] serve: a long conversation survives an engine restart - parked conversations saved at a graceful stop and loaded at the next start, to files or (Windows) kept in RAM with no disk writes (opt-in STRATA_PARK_DIR, STRATA_PARK_RAMLIKE=1; layer split included)
- #5476 [Engine] gfx103x decode: opt-in one-warp MMVQ for K <= 4096 and a per-format expert layout table (STRATA_MMVQ_NW1, STRATA_EXP_TABLE)
- #5475 [Engine] opt-in int8 copies for the persistent hc read (STRATA_HC_PERSIST_Q8, STRATA_HC_I8ONLY)
- #6394 [GUI] preserve long-running desktop agent sessions
- #3919 [Engine] kv: size the resident ring from the run's depth (opt-in, STRATA_KV_RING_ADAPT)
- #3648 [Engine] --peer-device host route: STRATA_PF_PEER_PIECES=1 (opt-in) moves the activations and the sums in 8 pieces (+7 % prompt on 2x V100)
- #2427 [Engine] HC_Q8: skip the layers' bf16 hyper-connection copies on a layer split too
- #6427 [Engine] layer split auto with DFlash2: plan DFlash2's drafter and after-cache bind on the last stage, and keep the gate-off fallback loadable
- #4766 [Engine] gfx906: build dflash2 for Radeon VII
- #6425 [Engine] q36: card-1 expert tier (--peer-device) for Qwen3.6-35B-A3B, 37.9 → 74.8 tok/s
- #4019 [Engine] Mac: calibrate the decode kernels' tiles on the device (setup.sh --calibrate)
- #3819 [Engine] Layer split: a stage's commit without the host wait, as on one GPU (`STRATA_SPLIT_COMMIT_ASYNC=1`, opt-in)
- #3945 [Engine] serve: a park moves the checkpoint buffers it already holds instead of copying them, same image (opt-in, `STRATA_CKPT_PARK_REUSE=1`)
- #3944 [Engine] --pipeline-windows: the front stages' GDN snapshot by buffer rotation instead of a copy per window, same bits (opt-in, `STRATA_PIPELINE_SNAP_RING=1`)
- #3760 [Engine] A third decode window in flight on a layer split (`--pipeline-windows 3`, opt-in): a host-side scheduler, a slot pool, per-stage undo, an 11-output draft chain
- #6288 [Engine] conversation cache SSD tier: a layer split's conversations and a kept folder across restarts (--conversation-spill-persist on)
- #6424 [Other] tests: isolate the two SYCL-path drafting tests from a real sibling Strata-data
- #4374 [Other] serve: opt-in "system_reminders": "tail" keeps a client's system-reminder blocks at the end of the prompt
- #6418 [Engine] Leave STRATA_BATCH_SELF_COMMIT out of the Linux CUDA parallel defaults when STRATA_PIPED_MTP is asked (#6160)
- #6414 [Engine] gfx906: reference build for MI50/MI60/Radeon VII (0.2.11 + int6 kernels)
- #5562 [Engine] prefill: the fused layout's shrink asks every layer's path, not only the formats
- #4883 [Other] Setup: --gpus on Windows AMD: RDNA 2, 3 and 4 cards sharing one model
- #6314 [Benchmark] RX 6700 XT and RX 6600 (x1, x2, with and without a rocBLAS table) / Qwen3.8-Flash-Next / IQ3_S / strata-qwen38-thinking-v1-draft.1 on the 0.2.16 rx6000 package
- #4859 [Engine] a 3-bit Lloyd-Max K/V (--kv oct3) with an exact fp16 window, and the gfx1100 decode ports
- #5494 [Other] Windows: recognise the Radeon Pro V620 (PCI id 0x73A1) as gfx1030
- #6249 [Engine] Opt-in STRATA_PREFILL_EQUAL_TAIL: keep a short last prompt chunk instead of a second streamed pass
- #6228 [Engine] split: Gyro trellis experts (TQ2_T, TQK6, TQK7) on Turing (sm_75), opt-in
- #6009 [Docs] speed projection: a request that switches reads its prompt again, the other setting's parked conversation stays (measured on 2x P100)
- #5986 [Engine] conversation cache: a layer split borrows a parked conversation's prefix (opt-in STRATA_SPLIT_BORROW=1, #5612; 2x P100: main agent's return 33,317 -> 2,672 tokens read)
- #5972 [Other] add opt-in Rust strata-fit crate for VRAM and multi-GPU fit math
- #5803 [Engine] gfx1152: opt-in thread-per-block long-prompt scorer (STRATA_SCORES_THREAD=1)
- #4867 [Engine] HIP/Linux: keep the kernel's NUMA balancing off the engine's memory (GPU queue evictions)
- #5599 [Docs] reproducible output: add the CPU prompt share and the route tail skip, and correct both rows (#4673)
- #5500 [Engine] L2 session cache: cut prefill time by ~95% on returning conversations
- #5432 [Engine] Do not force WindowInvariant CPU experts on DFlash2 windows when MTP fallback is loaded
- #4295 [Engine] gfx1151 hipBLASLt table for a system ROCm 7.2.4 (hipBLASLt 100202)
- #4175 [Engine] hot-swap the running model at runtime
- #4154 [Docs] ROCm 6.3.3 / gfx906 report — 2x MI50, system ROCm, build + run + 12 h split-config soak
- #3625 [Engine] release private checkpoint-copy streams when worker threads exit
- #3522 [Other] calibrate: keep the PCIe/draft-floor winner only when it beats this run's own noise instead of the fixed >3%
- #3404 [Engine] conversation cache: drop an older copy of the conversation rewritten turns back (#342 follow-up)
- #2115 [Other] setup: the SYCL setup on an integrated Arc (xe): native launcher, a RAM fit rule, no KV streaming
- #6149 [Engine] generate: skip post-search CUDA0 reload when stage_trim already trimmed (#6110)
- #6010 [Docs] conversation cache: what more clients than slots costs, with a measurement (eight agents on 2x P100: 792 s -> 454 s with 16 slots)
- #5873 [Engine] Gyro-L: 2.3x faster AVX2 CPU rows for the pair trellis (TQP5/6/7), bit-exact
- #5868 [Engine] Pipelined batch slots: a slot that reads alone sends its next window ahead (2x RTX 3090, chats taking turns: -31% first token, same output)
- #5866 [Engine] A slot alone in its group reads its turn in whole windows (2x RTX 3090: first token -4% to -15%, round total -2.8% at 440K, same output)
- #5830 [Engine] Unpack IQ4_NL PLE rows with AVX1
- #5748 [Engine] verify: STRATA_TAP_FILE works on a layer split
- #5588 [Engine] --ple-gguf: a side-loaded PLE table now wins the by-name resolution over the embedded copy
- #5540 [Engine] sycl: count helper/peer caches before running the adaptive tier (helper mode decode 32 -> 61 tok/s)