#5939 [Engine] Windows CUDA12 0.1.59: Tesla V100 GPU is lost during single- and dual-GPU inference
open · @cain473-cmd · 4 コメント · GitHub で見る
Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
本文
### Area [Engine] CUDA / GPU device loss ### Desktop app, Terminal or source? Strata Terminal with saved launch presets on Windows 11 Pro. ### What happened The NVIDIA Tesla V100 becomes inaccessible to NVML during Strata inference, and `nvidia-smi` shows `GPU is lost`. The RTX 4070 Ti remains visible. This occurred in both a dual-GPU run and a separate **V100-only** run, so a cross-GPU layer split is not required for reproduction. Subsequent engine restart attempts cannot find a CUDA-capable GPU. ### Smallest reproduction steps known so far 1. Windows 11 Pro, NVIDIA driver 576.57, Strata engine 0.1.59 (Windows CUDA12), Qwen3.8-Flash-Next IQ3_S, maximum context 262144, KV int8 / resident 32768, 26 GiB RAM-resident experts and the ESP control vector. 2. Run Strata with RTX 4070 Ti + Tesla V100 (layer split 27), or use **only the V100** with split disabled and MTP model disabled. 3. Send inference requests. The failure is intermittent, not every request. It has been observed with about 60K tokens (user observation; trace not included) and after longer requests. Do not interpret ~160K tokens as a proven minimum threshold. 4. V100 stops responding to NVML and remains lost until recovery outside Strata. RTX 4070 Ti remains accessible. **Expected:** V100 stays accessible and inference continues, or Strata reports and handles a recoverable CUDA error. **Actual:** `nvidia-smi` shows `GPU is lost`; the V100-side `strata.exe` process disappears. A restart after the 15:34 incident reports `native embedding: cannot pin 322 MiB: no CUDA-capable device is detected, nor place it in VRAM`. ### Reproduction status Reproduced locally in two different Strata GPU layouts. Root cause unknown. ### Strata version Terminal 0.2.12; engine 0.1.59 (Windows x86_64 CUDA12, CUDA 12.9). GPU driver 576.57. ### Graphics card, memory and system - Windows 11 Pro; AMD Ryzen 9 7900; 64 GB RAM - ASUS ProArt X870E-CREATOR WiFi - GPU 0: RTX 4070 Ti 12 GB (WDDM, display) - GPU 1: Tesla V100-SXM2-16GB, sm_70 (TCC, PCIe adapter) - SilverStone DA750R Gold 750 W PSU (user reported) ### Model, size and settings - Qwen3.8-Flash-Next IQ3_S, about 84 GB on disk - `--max-context 262144 --kv int8 --kv-resident 32768 --resident-budget-gib 26 --vram-reserve-mib 1536` - ESP control vector (project, layers 4-44) enabled for both runs - Dual: layer split 27, MTP enabled - Solo: V100 selected, layer split off, MTP model off, conversation parking off; **`--spec 4` and suffix drafting were still active in the engine despite an earlier CLI-level 'drafting off' plan** ### Evidence and timeline (2026-11-14 JST, UTC+09:00) **Dual run:** V100 became lost around 10:52. Earlier engine output includes a successfully completed ~156K-token prompt, then an engine restart at 10:52:36. This does not establish the exact operation that caused the failure. **V100-only run:** - 15:29:05 engine started on V100, 262144-token context confirmed in engine argv. - Engine initialized 4520 resident expert slots (8.62 GiB GPU cache), 26 GiB host RAM complement, and KV streaming. - The log reports a 161432-token prompt completed with 527 generated tokens; after that no explicit CUDA error was recorded before the engine restart. - 15:34:32 last normal nvidia-smi sample: V100 56C, 71W, 94% GPU utilization, 14944MiB VRAM allocated. - 15:34:34 first `GPU is lost` sample; RTX 4070 Ti still visible. - 15:34:59 and later engine restart attempts: `no CUDA-capable device is detected`. These samples **do not prove** an overheating cause (temperature was modest) nor rule out transient electrical/PCIe faults (the power readings are samples, not peak transients). ### Independent GPU stress-test evidence The same V100 was exercised outside Strata using NVIDIA GPUStressTest and CUDA tests. A first extended run was interrupted by the test's watchdog (FP64 was marked FAILED), while a subsequent extended run logged PASS for all five stress test types using 50 loops per type. Stress-test monitoring also included high VRAM consumption and approximately 300 W power observations without the `GPU is lost` state. Thus sustained compute load alone did not reproduce this failure, but these results **do not exclude** GPU, power, adapter or driver problems. ### What I need help with - Is this a known Volta sm_70 / CUDA12 issue in engine 0.1.59? - Can maintainers identify useful tracing for expert-cache swapping, KV streaming, suffix-draft verification or CUDA synchronization immediately before NVML device loss? - Is there a suggested minimally invasive diagnostic to distinguish engine execution from driver, PCIe adapter or transient power problems? ### Logs The following staged **filtered and redacted diagnostic artifacts** can be shared through the evidence link after review. Full raw originals remain on my local machine and are not automatically published. - `conversation-v100only-262k-gpulost-monitor-excerpt.txt` - `host-snapshot.txt` - `source-00-iq3s-dual-128k-esp.engine.log.txt` - `source-01-iq3s-dual-128k-safe.engine.log.txt` - `source-02-iq3s-dual-128k.engine.log.txt` - `source-03-iq3s-dual-262k-ram22-esp.engine.log.txt` - `source-04-iq3s-dual-262k-ram24-esp.engine.log.txt` - `source-05-iq3s-dual-262k-ram26-esp.engine.log.txt` - `source-06-iq3s-dual-262k-ram28-esp.engine.log.txt` - `source-07-iq3s-dual-64k.engine.log.txt` - `source-08-iq3s-dual-resident.engine.log.txt` - `source-09-iq3s-dual-safe.engine.log.txt` - `source-10-iq3s-dual-trim.engine.log.txt` - `source-11-iq3s-v100only-262k-ram26-esp.engine.log.txt` - `source-12-iq3s-v100only-32k-ram26-esp.engine.log.txt` - `source-13-gst-long-20261114-125416-summary.txt.txt` - `source-14-gst-long-20261114-125416.err.txt.txt` - `source-15-gst-long-20261114-125416.log.txt` - `source-16-gst-long-20261114-130013-summary.txt.txt` - `source-17-gst-long-20261114-130013.err.txt.txt` - `source-18-gst-long-20261114-130013.log.txt` - `source-19-gst-long-20261114-134058-summary.txt.txt` - `source-20-gst-long-20261114-134058.err.txt.txt` - `source-21-gst-long-20261114-134058.log.txt` - `source-22-gst-short-20261114-122937.log.txt` - `source-23-v100-long-monitor-20261114-125416.csv.txt` - `source-24-v100-long-monitor-20261114-130013.csv.txt` - `source-25-v100-long-monitor-20261114-134058.csv.txt` - `source-26-v100-monitor-20261114-122937.csv.txt` - `source-27-v100-monitor-20261114-122937.err.txt.txt` - `source-28-v100-power-test_back1.csv.txt` - `source-29-nvidia-setup-diagnostic.log.txt` - `source-30-strata-0910-failure-events.txt.txt` - `source-31-strata-diagnostic.txt.txt` - `source-32-strata-error-20261114-092950.txt.txt` - `source-33-strata-failure-details.txt.txt` - `source-34-strata-gpu-watch-20261114-092408.log.txt` - `source-35-strata-gpulost-full-20261114.log.txt` - `source-36-__1_20261114_110436.txt.txt` - `windows-event-summary.txt` **Evidence link:** https://gist.github.com/cain473-cmd/cc11c8e32f6afdd43159a77387e69f02
関連リンク
インストール・モデル・リリースへのサイト内リンク。