#5990 [Engine] V100 (sm_70) on Linux: 13× Xid 79 + PCIe CmpltTO across engines 0.1.35–0.1.59.1 — not version-specific
open · @Argon2400 · 5 comentários · No GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux
Descrição
> **Updated 2026-11-15** with the full evidence bundle: the PCIe topology on this board (a PLX PEX 8747 switch sits between the CPU root port and both cards), a per-event AER component matrix derived from all 13 kernel captures, and **corrected telemetry numbers** — an earlier revision of this report quoted "57–127 W", but the attached 1 Hz telemetry shows higher values (up to 263.20 W in one event). See "Conditions measured at failure time" and "Evidence bundle".
### Area
[Engine] CUDA / PCIe device loss on Linux
### Desktop app, Terminal or source?
Source (setup.sh or a build) — a self-built source install (`STRATA_EXPERIMENTAL_SM60=1`, sm_70 cubin only, no PTX, conda gcc 12.3). Not the Desktop app, not a Terminal binary, not Docker.
### What happened
Since 2026-10-03 the engine has lost a Tesla V100-SXM2-16GB from the PCIe bus 13 times on this machine. The fatal part of the signature is identical every time:
```
pcieport 0000:00:03.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
pcieport 0000:00:03.0: device [8086:2f08] error status/mask=00004000/00000000
pcieport 0000:00:03.0: [14] CmpltTO (First)
pcieport 0000:00:03.0: AER: Uncorrectable (Non-Fatal) error message received from 0000:00:03.0
nvidia 0000:06:00.0: AER: can't recover (no error_detected callback)
pcieport 0000:00:03.0: AER: device recovery failed
NVRM: Xid (PCI:0000:06:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:06:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
```
The PCIe completion timeout at the CPU root port `0000:00:03.0` (`8086:2f08`) escalates to an unrecoverable AER event, the card drops off the bus with Xid 79 + 154, and the engine's own log ends with:
```
[2026-10-03 01:52:39] [strata] the engine reported an error: verify: timed out at layer 5;
its GPU waits were released but the GPU did not finish within 5 s (#267)
[2026-11-14 23:46:59] [strata] the engine stopped unexpectedly (exit code None). Unrecoverable
native verification failure: verify: timed out at layer 46;
its GPU waits were released but the GPU did not finish within 5 s (#267)
```
Throughput collapses first (e.g. 93 → 63 → 11 tok/s), the request ends with `error`/`cancel`, `nvidia-smi` no longer lists the card, and the engine process becomes unkillable (`kill -9` does not free the VRAM). Recovery needs a **cold power cycle** (shutdown + mains off ~30 s); a warm `reboot` was once not enough and the card did not re-enumerate.
**Every one of the 13 events also carries a physical-layer receive error on the PCIe switch port that feeds the card that failed** — and only on that port:
```
pcieport 0000:00:03.0: AER: Multiple Correctable error message received from 0000:04:08.0
pcieport 0000:04:08.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
pcieport 0000:04:08.0: device [10b5:8747] error status/mask=00000001/0000e000
pcieport 0000:04:08.0: [ 0] RxErr (First)
pcieport 0000:04:08.0: [20] UnsupReq (First)
```
`04:08.0` is the PLX PEX 8747 downstream port that feeds GPU1; in the two October events (GPU0) it is `04:10.0`. This is a *correctable* error, so it is not the fatal event — but a physical-layer receive error that shows up on the failing card's own link, in all 13 events, and on no other port, is consistent with a marginal link or carrier.
**Expected:** either the inference completes, or the engine reports a recoverable CUDA/PCIe error and the card stays on the bus (a driver-level reset, not a bus drop).
**Actual:** the completion timeout escalates to an unrecoverable AER event, the GPU falls off the bus, and the machine must be power-cycled. Any in-flight request is lost.
**This is not version-specific.** The 13 events span six upstream engines (0.1.35, 0.1.40.2, 0.1.54, 0.1.57, 0.1.58, 0.1.59.1) and one third-party fork (0.1.45.7). Three of them — two of the last four — happened on the project's own "known-good" rollback build 0.1.54 / app 0.2.8, so the earlier reading that 0.1.54 is clean does not hold here. 11 of the 13 events are the same card (GPU1, `0000:06:00.0`, root port `0000:00:03.0`); the first two are its sibling (GPU0, `0000:05:00.0`) with the same signature.
**Related threads — not a duplicate claim.** #4085 is the same signature (V100, Xid 79 + PCIe CmpltTO) reported from a different reporter on a different machine; its author withdrew the version-regression hypothesis and re-classified it as hardware. #5939 is a V100 device loss during inference on Windows. Per CONTRIBUTING.md "Before starting" rule 1 this is filed as new evidence rather than silently duplicated — if maintainers prefer a single thread, please merge this into #4085 or redirect it; we are asking for that decision, not competing with either thread.
### Smallest steps or command that reproduce it
We **cannot reproduce it on demand** — nothing in the engine's inputs reliably triggers it. The steps below reproduce the *observation*, not the root cause:
1. Start the engine on 2× V100-SXM2-16GB (sm_70, no NVLink), 2-GPU layer split (auto), `--max-context 262144`.
2. Send ordinary chat requests (256-token generations), or run a long-context prefill.
3. At an unpredictable point, decode/prefill throughput collapses (e.g. 93 → 63 → 11 tok/s), the request ends with `error`/`cancel`, and `nvidia-smi` no longer lists the card.
4. Recovery requires a cold power cycle (shutdown + mains off ~30 s).
The only deterministic sequence we ever had was on 2026-11-08: **4/4 attempts** with `UD-IQ3_XXS` and a 230K-token prompt fell off the bus at the same root port. That model was subsequently retired, and the identical signature later occurred with a 1.2K-token decode, so "long prefill" is a trigger amplifier, not a requirement.
### Reproduction status
Reproduced locally
The failure was observed 13 times on this machine between 2026-10-03 and 2026-11-14, with a kernel capture for every occurrence (see "Evidence bundle"). It is intermittent and cannot be triggered on demand; we did not observe it in a debugger, so the root cause is not established.
### Strata version
App 0.2.8 / 0.2.11 / 0.2.12 / 0.2.13.1 · engines 0.1.35, 0.1.40.2, 0.1.54, 0.1.57, 0.1.58, 0.1.59.1, plus a third-party fork (0.1.45.7, `jmnargi` ee96b35e). All self-built from source.
### Graphics card, memory and system
2× Tesla V100-SXM2-16GB (sm_70, 16 GB each, no NVLink) · failing card GPU1 `0000:06:00.0`, serial `0324218161xxx` · 125 GiB DDR4-2133 (8×16 GB) · ASUS X99-E WS, Xeon E5-2678 v3 (12C/24T) · Fedora 44, kernel `7.0.12-201.fc44.x86_64` · NVIDIA driver 580.159.04, CUDA 12.8 (`/usr/local/cuda-12.8`) · PCIe Gen3 x16 on both cards · both GPUs nominally power-capped at 250 W (see the correction in "Conditions measured at failure time").
### Model, size and settings
Qwen3.8-Flash-Next IQ3_XXS (Swift 1.5 SC117; ALESHA + cvec 4–44), UD-IQ3_XXS, Coder IQ1_M · 2-GPU layer split (auto) · `--max-context 262144` · `--kv int8` · `--prefill 32768` / auto · `--vram-reserve-mib 700` (661 on the older configs).
### Environment
| Item | Value |
|---|---|
| GPU | 2× Tesla V100-SXM2-16GB, sm_70, no NVLink; failing card **GPU1 `0000:06:00.0`**, serial `0324218161xxx` |
| Driver / CUDA | NVIDIA 580.159.04 / CUDA 12.8 (`/usr/local/cuda-12.8`) |
| OS / kernel | Fedora 44, kernel `7.0.12-201.fc44.x86_64` |
| CPU / board / RAM | Xeon E5-2678 v3 (12C/24T), ASUS **X99-E WS**, 125 GiB DDR4-2133 |
| PCIe topology | `00:03.0` (CPU root port 3) → `04:00.0` **PLX PEX 8747** switch (`10b5:8747`) → `04:08.0` → **GPU1 `06:00.0`**, and `04:10.0` → **GPU0 `05:00.0`** (both V100s sit behind the same PLX switch) |
| Root port ID | `8086:6f08` for E01/E02 (i7-6850K), `8086:2f08` after the 2026-11-06 CPU swap (Xeon E5-2678 v3); the rest of the signature is unchanged |
| Build | source build, `STRATA_EXPERIMENTAL_SM60=1`, sm_70 cubin only (no PTX), conda gcc 12.3 |
| Strata versions | 0.1.35 → 0.1.40.2 → 0.1.43.1 → 0.1.46.1 → **0.1.54 (app 0.2.8)** → 0.1.57 (app 0.2.11) → 0.1.58 (app 0.2.12) → 0.1.59.1 (app 0.2.13.1); plus fork 0.1.45.7 |
| Models | Qwen3.8-Flash-Next IQ3_XXS (Swift 1.5 SC117; ALESHA + cvec 4–44), UD-IQ3_XXS, Coder IQ1_M |
| Config | 2-GPU layer split (auto), `--max-context 262144`, `--kv int8`, `--prefill 32768`/auto, `--vram-reserve-mib 700` (661 on older configs) |
| Power | **both GPUs nominally capped at 250 W** (`gpu-power-limit.service`, enabled; first applied 2026-11-08 16:45, re-applied every boot) — but see the measured value in "Conditions measured at failure time" |
| Misc | `gpu-xid-watch.service` active; EDAC CE/UE = 0; PCIe link Gen3 x16 on both cards |
### Event list (all 13 occurrences, local time HKT)
| # | Time | GPU | Strata app / engine | Model | Workload at failure |
|---|---|---|---|---|---|
| 1 | 2026-10-03 01:52:15 | GPU0 `05:00.0` | 0.1.35 | Coder IQ1_M | decode/thinking at 66 tok/s → stall; retry on a 53-token prompt then stalled |
| 2 | 2026-10-08 23:52:03 | GPU0 `05:00.0` | 0.1.40.2 | Swift SC117 IQ3_XXS | 632 tok / 42 s → error |
| 3 | 2026-11-08 14:25:07 | GPU1 `06:00.0` | app 0.2.11 / 0.1.57 | UD-IQ3_XXS | 230K prefill benchmark |
| 4 | 2026-11-08 16:28:20 | GPU1 | app 0.2.11 / 0.1.57 | UD-IQ3_XXS | 230K prefill benchmark |
| 5 | 2026-11-08 16:56:50 | GPU1 | app 0.2.11 / 0.1.57 | UD-IQ3_XXS | 230K prefill benchmark (after the 250 W cap was applied) |
| 6 | 2026-11-08 17:53:02 | GPU1 | app 0.2.11 / 0.1.57 | UD-IQ3_XXS | 230K prefill benchmark (after the 250 W cap was applied) |
| 7 | 2026-11-09 15:15:02 | GPU1 | app 0.2.12 / 0.1.58 | ALESHA IQ3_XXS + cvec | ~31K prompt |
| 8 | 2026-11-09 18:26:23 | GPU1 | **app 0.2.8 / 0.1.54** | Swift SC117 IQ3_XXS | 2,146-token prompt, decode → error |
| 9 | 2026-11-10 11:36:05 | GPU1 | app 0.2.13.1 / 0.1.59.1 | ALESHA IQ3_XXS + cvec | 2,085-token prompt, decode → error |
| 10 | 2026-11-10 13:19:04 | GPU1 | **app 0.2.8 / 0.1.54** | Swift SC117 IQ3_XXS | 1,238-token prompt, thinking ~90 tok/s |
| 11 | 2026-11-14 18:23:25 | GPU1 | **app 0.2.8 / 0.1.54** | Swift SC117 IQ3_XXS | 256-token decode 50–92 tok/s → stall |
| 12 | 2026-11-14 19:59:33 | GPU1 | fork `jmnargi ee96b35e` / 0.1.45.7 | Swift SC117 IQ3_XXS | 256-token decode 51–94 tok/s → stall |
| 13 | 2026-11-14 23:46:29 | GPU1 | fork F3 / 0.1.45.7 | Swift SC117 IQ3_XXS | 1,164-token prompt, thinking 83 tok/s → stall |
Counts by engine: 0.1.35 ×1 · 0.1.40.2 ×1 · 0.1.57 ×4 · 0.1.58 ×1 · **0.1.54 ×3** · 0.1.59.1 ×1 · fork 0.1.45.7 ×2.
### Exposure windows by engine
| Engine | Window | Events |
|---|---|---|
| 0.1.43.1 → 0.1.46.1 | 2026-10-09 → 10-20 (11 d) | 0 |
| 0.1.46.1 | 10-20 → 10-30 (10 d) | 0 |
| **0.1.54** | 10-30 20:07 → 11-06 11:22 (~6.6 d) | 0 |
| 0.1.57 | 11-06 11:22 → 11-08 19:25 (~2.3 d) | 4 |
| 0.1.58 | 11-08 19:25 → 11-09 16:19 (~21 h) | 1 |
| **0.1.54** (rollback, 2nd window) | 11-09 16:19 → 11-10 07:01; 11-10 11:53 → 13:29; 11-14 18:16 → 18:24 | **3** |
| 0.1.59.1 | 11-10 07:01 → 11:52 (4.7 h) | 1 |
| fork 0.1.45.7 | 11-14 19:37 → 23:57 | 2 |
GPU0 failed only **before** 2026-10-09 (twice, on 0.1.35 / 0.1.40.2); GPU1 failed only from **2026-11-08** onward.
### Per-event AER component matrix
Counted from the kernel capture of each event (`kernel/E*.txt` in the bundle):
| Event | CmpltTO (root port) | RxErr (PLX port of the failed card) | UnsupReq (same PLX port) | `dev_put` WARN |
|---|---|---|---|---|
| E01 | 10 | 1 × `04:10.0` | 2 | 0 |
| E02 | 10 | 1 × `04:10.0` | 2 | 0 |
| E03 | 9 | 1 × `04:08.0` | 1 | 0 |
| E04 | 11 | 1 × `04:08.0` | 3 | 0 |
| E05 | 10 | 1 × `04:08.0` | 2 | 0 |
| E06 | 21 | 1 × `04:08.0` | 12 | 6 |
| E07 | 11 | 1 × `04:08.0` | 2 | 0 |
| E08 | 11 | 1 × `04:08.0` | 2 | 0 |
| E09 | 11 | 1 × `04:08.0` | 2 | 0 |
| E10 | 9 | 1 × `04:08.0` | 1 | 0 |
| E11 | 22 | 1 × `04:08.0` | 13 | 8 |
| E12 | 21 | 1 × `04:08.0` | 12 | 6 |
| E13 | 11 | 1 × `04:08.0` | 2 | 0 |
Two things thNo site
Links install, modelos, releases.