#5939 [Engine] Windows CUDA12 0.1.59: Tesla V100 GPU is lost during single- and dual-GPU inference

open · @cain473-cmd · 4 comentarios · En GitHub

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Descripción

### Area
[Engine] CUDA / GPU device loss

### Desktop app, Terminal or source?
Strata Terminal with saved launch presets on Windows 11 Pro.

### What happened
The NVIDIA Tesla V100 becomes inaccessible to NVML during Strata inference, and `nvidia-smi` shows `GPU is lost`. The RTX 4070 Ti remains visible. This occurred in both a dual-GPU run and a separate **V100-only** run, so a cross-GPU layer split is not required for reproduction. Subsequent engine restart attempts cannot find a CUDA-capable GPU.

### Smallest reproduction steps known so far
1. Windows 11 Pro, NVIDIA driver 576.57, Strata engine 0.1.59 (Windows CUDA12), Qwen3.8-Flash-Next IQ3_S, maximum context 262144, KV int8 / resident 32768, 26 GiB RAM-resident experts and the ESP control vector.
2. Run Strata with RTX 4070 Ti + Tesla V100 (layer split 27), or use **only the V100** with split disabled and MTP model disabled.
3. Send inference requests. The failure is intermittent, not every request. It has been observed with about 60K tokens (user observation; trace not included) and after longer requests. Do not interpret ~160K tokens as a proven minimum threshold.
4. V100 stops responding to NVML and remains lost until recovery outside Strata. RTX 4070 Ti remains accessible.

**Expected:** V100 stays accessible and inference continues, or Strata reports and handles a recoverable CUDA error.

**Actual:** `nvidia-smi` shows `GPU is lost`; the V100-side `strata.exe` process disappears. A restart after the 15:34 incident reports `native embedding: cannot pin 322 MiB: no CUDA-capable device is detected, nor place it in VRAM`.

### Reproduction status
Reproduced locally in two different Strata GPU layouts. Root cause unknown.

### Strata version
Terminal 0.2.12; engine 0.1.59 (Windows x86_64 CUDA12, CUDA 12.9). GPU driver 576.57.

### Graphics card, memory and system
- Windows 11 Pro; AMD Ryzen 9 7900; 64 GB RAM
- ASUS ProArt X870E-CREATOR WiFi
- GPU 0: RTX 4070 Ti 12 GB (WDDM, display)
- GPU 1: Tesla V100-SXM2-16GB, sm_70 (TCC, PCIe adapter)
- SilverStone DA750R Gold 750 W PSU (user reported)

### Model, size and settings
- Qwen3.8-Flash-Next IQ3_S, about 84 GB on disk
- `--max-context 262144 --kv int8 --kv-resident 32768 --resident-budget-gib 26 --vram-reserve-mib 1536`
- ESP control vector (project, layers 4-44) enabled for both runs
- Dual: layer split 27, MTP enabled
- Solo: V100 selected, layer split off, MTP model off, conversation parking off; **`--spec 4` and suffix drafting were still active in the engine despite an earlier CLI-level 'drafting off' plan**

### Evidence and timeline (2026-11-14 JST, UTC+09:00)
**Dual run:** V100 became lost around 10:52. Earlier engine output includes a successfully completed ~156K-token prompt, then an engine restart at 10:52:36. This does not establish the exact operation that caused the failure.

**V100-only run:**
- 15:29:05 engine started on V100, 262144-token context confirmed in engine argv.
- Engine initialized 4520 resident expert slots (8.62 GiB GPU cache), 26 GiB host RAM complement, and KV streaming.
- The log reports a 161432-token prompt completed with 527 generated tokens; after that no explicit CUDA error was recorded before the engine restart.
- 15:34:32 last normal nvidia-smi sample: V100 56C, 71W, 94% GPU utilization, 14944MiB VRAM allocated.
- 15:34:34 first `GPU is lost` sample; RTX 4070 Ti still visible.
- 15:34:59 and later engine restart attempts: `no CUDA-capable device is detected`.

These samples **do not prove** an overheating cause (temperature was modest) nor rule out transient electrical/PCIe faults (the power readings are samples, not peak transients).

### Independent GPU stress-test evidence
The same V100 was exercised outside Strata using NVIDIA GPUStressTest and CUDA tests. A first extended run was interrupted by the test's watchdog (FP64 was marked FAILED), while a subsequent extended run logged PASS for all five stress test types using 50 loops per type. Stress-test monitoring also included high VRAM consumption and approximately 300 W power observations without the `GPU is lost` state. Thus sustained compute load alone did not reproduce this failure, but these results **do not exclude** GPU, power, adapter or driver problems.

### What I need help with
- Is this a known Volta sm_70 / CUDA12 issue in engine 0.1.59?
- Can maintainers identify useful tracing for expert-cache swapping, KV streaming, suffix-draft verification or CUDA synchronization immediately before NVML device loss?
- Is there a suggested minimally invasive diagnostic to distinguish engine execution from driver, PCIe adapter or transient power problems?

### Logs
The following staged **filtered and redacted diagnostic artifacts** can be shared through the evidence link after review. Full raw originals remain on my local machine and are not automatically published.

- `conversation-v100only-262k-gpulost-monitor-excerpt.txt`
- `host-snapshot.txt`
- `source-00-iq3s-dual-128k-esp.engine.log.txt`
- `source-01-iq3s-dual-128k-safe.engine.log.txt`
- `source-02-iq3s-dual-128k.engine.log.txt`
- `source-03-iq3s-dual-262k-ram22-esp.engine.log.txt`
- `source-04-iq3s-dual-262k-ram24-esp.engine.log.txt`
- `source-05-iq3s-dual-262k-ram26-esp.engine.log.txt`
- `source-06-iq3s-dual-262k-ram28-esp.engine.log.txt`
- `source-07-iq3s-dual-64k.engine.log.txt`
- `source-08-iq3s-dual-resident.engine.log.txt`
- `source-09-iq3s-dual-safe.engine.log.txt`
- `source-10-iq3s-dual-trim.engine.log.txt`
- `source-11-iq3s-v100only-262k-ram26-esp.engine.log.txt`
- `source-12-iq3s-v100only-32k-ram26-esp.engine.log.txt`
- `source-13-gst-long-20261114-125416-summary.txt.txt`
- `source-14-gst-long-20261114-125416.err.txt.txt`
- `source-15-gst-long-20261114-125416.log.txt`
- `source-16-gst-long-20261114-130013-summary.txt.txt`
- `source-17-gst-long-20261114-130013.err.txt.txt`
- `source-18-gst-long-20261114-130013.log.txt`
- `source-19-gst-long-20261114-134058-summary.txt.txt`
- `source-20-gst-long-20261114-134058.err.txt.txt`
- `source-21-gst-long-20261114-134058.log.txt`
- `source-22-gst-short-20261114-122937.log.txt`
- `source-23-v100-long-monitor-20261114-125416.csv.txt`
- `source-24-v100-long-monitor-20261114-130013.csv.txt`
- `source-25-v100-long-monitor-20261114-134058.csv.txt`
- `source-26-v100-monitor-20261114-122937.csv.txt`
- `source-27-v100-monitor-20261114-122937.err.txt.txt`
- `source-28-v100-power-test_back1.csv.txt`
- `source-29-nvidia-setup-diagnostic.log.txt`
- `source-30-strata-0910-failure-events.txt.txt`
- `source-31-strata-diagnostic.txt.txt`
- `source-32-strata-error-20261114-092950.txt.txt`
- `source-33-strata-failure-details.txt.txt`
- `source-34-strata-gpu-watch-20261114-092408.log.txt`
- `source-35-strata-gpulost-full-20261114.log.txt`
- `source-36-__1_20261114_110436.txt.txt`
- `windows-event-summary.txt`

**Evidence link:** https://gist.github.com/cain473-cmd/cc11c8e32f6afdd43159a77387e69f02

En el sitio

Enlaces a install, modelos, releases.