#6631 [Engine] Multi-GPU: Inkling 2 Mini loads on a single GPU only (main/helper setting ignored), crashes when context fills VRAM
open · @AswinPG · 2 commentaires · Sur GitHub
Setup & installMulti-GPUNVIDIA / CUDAModels & quantsWindows
Description
## Description With two GPUs (RTX 5080 + RTX 5070 Ti), Inkling 2 Mini (15 GB quant) in Strata loads onto only one GPU instead of being split across both. The "main GPU / helper GPU" setting doesn't seem to be respected. Because only one card's VRAM is used, the usable context length is very small, and Strata crashes once the KV cache fills the remaining VRAM. ## Environment - Strata version: latest release 0.2.19 - OS: Windows 11 - NVIDIA driver / CUDA version: [driver, CUDA] - GPU 1: RTX 5080 (16 GB) - GPU 2: RTX 5070 Ti (16 GB) - System RAM: 32 GB DDR5 - Model: Inkling 2 Mini, 15 GB file ([quant name, e.g. Q4_K_M, if known]) ## Steps to reproduce 1. Install Strata with both GPUs enabled. 2. Set the RTX 5080 as the main GPU and the RTX 5070 Ti as the helper GPU. 3. Load Inkling 2 Mini (15 GB). 4. Check VRAM usage on both GPUs (nvidia-smi or Task Manager). ## Expected behavior - The model is split across both GPUs (or at minimum loads on the 5080 as main, with the 5070 Ti used for overflow layers and KV cache). - The combined ~32 GB VRAM allows a much larger context window. ## Actual behavior - With 5080 = main and 5070 Ti = helper, the model loads **only on the 5070 Ti**. The 5080 is not used for the model. - If I disable the 5070 Ti, the model loads on the 5080 only. - In both cases only one GPU's VRAM is used. Context length stays low regardless of the setting I choose. - Once context fills the remaining VRAM on that single GPU, Strata crashes. Nothing spills over to the second GPU. - The behavior is the same on every Strata version I tried, including the latest. ## Additional notes - The model file (15 GB) should fit comfortably across 2x 16 GB cards, so this doesn't look like a genuine out-of-memory limit. - The main/helper assignment appears to be inverted or ignored, since the "helper" card is the one that gets the model. - No tensor split or layer split seems to be applied. ## Questions - Is there a setting for tensor split / layer offload per GPU that I'm missing? - Is multi-GPU splitting supported for this model or backend? - Is the main/helper ordering expected to follow CUDA device index rather than my selection?
Sur le site
Liens install, modèles, releases.