#5654 [Engine] Dual RTX 5060 Ti over OCuLink: Xid 79 during tune auto (0.1.58, Linux driver 615.71.09 open)

open · @ryu53293-netizen · 2 comentarios · En GitHub

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsSecurityWindowsLinux

Descripción

### Area
[Engine] (engine/server; observed during the official Terminal tuner)

### Desktop app, Terminal or source?
Terminal (official release), using an explicit existing run-config JSON.

### Reproduction status
Reproduced locally: one actual attempt on the current model/engine/driver, after a successful synthetic protection check. Two older incidents are historical context, not same-configuration repeats.

### Strata version
Terminal 0.2.12 / Control 0.1.58 / engine 0.1.58, build `8a42a9457db6+cuda13.0`. This report concerns that tested build; no claim about a later build that has not been run.

## Observed behavior
At 2026-11-09 10:33:34 UTC, NVIDIA reported Xid 79 (GPU has fallen off the bus) on PCI 0000:01:00.0, followed by Xid 154 / OS Reboot recovery actions on both GPUs. Strata reported:
  verify: layer 3 never rang (unspecified launch failure)
The GPU then remained unavailable. The external guard stopped the tuner (SIGTERM). No automatic retry, GPU reset, module reload or reboot was performed.

## Environment
GPD WIN Max 2 / Ryzen 7 7840U / 64 GB RAM (owner-provided hardware context).
Two RTX 5060 Ti 16 GB, one via native OCuLink and one via M.2-to-OCuLink (owner-provided wiring; physical connector-to-PCI mapping unverified).
Ubuntu 24.04; kernel 6.8.0-147-generic.
Loaded NVIDIA driver: 615.71.09 Open Kernel Module; DKMS installation (kernel trace and package audit included).
Terminal 0.2.12 / Control 0.1.58 / engine 0.1.58, build 8a42a9457db6+cuda13.0.
Model: Qwen3.8-Flash-Next, Unsloth UD-IQ4_XS-v31, four GGUF shards. Prepared using the bundled iq_pack.py --compat-bf16. Experts and PLE source bytes were not rewritten; small compatibility tensors include rounding conversions.

## Exact trigger / reproduction
See evidence/actual/command.json for the complete official CLI command and config/base.json for all model arguments and environment overrides. It invokes:
  strata tune auto <base.json> --mode quick --time 20 --use chat --no-save --name iq4xs-v31-44H8-4K --work-dir <new work dir> --status-file <progress.json> --gpu-lock <new lock> --boot-timeout 600 --output json
Important model settings: context4096, prefill16384, resident44GiB, headroom8GiB, GPU0/1 in PCI_BUS_ID order, layer split28, split device1, VRAM reserves1536/1100MiB, pinned resident cache, MTP and suffix draft enabled. Full values in config; no claim that the newer setup defaults are in use.

## Execution sequence
10:32:02 UTC: protected actual attempt started (before this, a synthetic CPU protection self-test passed; it did not load a model).
10:32:04 UTC: engine process started.
Engine session initialized and completed a 64-token warmup.
Two subsequent 200-token decode measurements completed.
The following measurement ended with GPU error after 167 generated tokens; full baseline and candidate comparison were incomplete.
10:33:34 UTC: Xid79, PCI01:00.0; then Xid154 on both GPUs.
Approximately 60 seconds of bounded cleanup followed. Total supervised actual attempt: 152.538 seconds including cleanup, not 152 seconds of stable inference.
No preset was saved or applied.

## System protection and observations
External user systemd cgroup: memory.high56GiB, memory.max58GiB, memory.swap.max0; CPUQuota1200%, TasksMax384, Nice10; parent heartbeat lease, runtime bound and PID/start-time/pidfd-owned cleanup. Host/GPU polling every250ms (extended GPU metrics every1s).
Sampled cgroup peak54.152GiB; memory.events high/max/oom/oom_kill all0.
Minimum host MemAvailable8.042GiB; minimum sampled GPU free1228.06MiB / 802.06MiB.
Maximum sampled GPU temperatures50C / 53C.
Cgroup swap peak0. Host swap-used rose from0 to256KiB; this was below the guard's1MiB swap stop threshold. This is not a zero-host-swap claim.
No configured low-memory, overtemperature or severe-reclaim stop was reported before GPU loss. Sampling does not establish the instantaneous condition at failure or exclude PCIe/power/driver faults.
The first recorded guard stop was NVML15: GPU is lost.

## Cleanup and evidence limitations
All owned processes and zombies exited; the cgroup was empty. cleanup_verified=false refers to GPU resource-release verification: telemetry became unavailable on the lost card, so GPU memory release could not be verified. It does not mean the tuner was left running.
The raw progress.json still says running because the tuner was interrupted; it is a stale snapshot. Use actual/result.json and cleanup.json for final status.
Post-run GPU audit saw only PCI02:00.0 (2MiB used), while PCI01:00.0 was unavailable. Post-run PCI01 had Memory/BusMaster disabled; full PCI capabilities were inaccessible to the non-root collector. Driver crash-dump/private privileged diagnostics are not included.
Actual engine /proc mappings were captured (PID3983): libcuda.so.615.71.09 plus libcudart.so.13, libcublas.so.13 and libcublasLt.so.13 from the selected0.1.58 CUDA13 engine directory. See evidence/ANALYSIS.json and library-maps for process identity and timestamps. CUDA13.0 is a build identifier, not a claim that all individual library patch versions were queried.

## Historical context / attribution boundary
On 2026-10-21 two attempts also reported Xid79 at PCI01:00.0 with driver580.178.04 and engine0.1.46, using a different NVFP4/GPTQ configuration. This run changed model, engine and driver, so it is not a controlled driver-only comparison and does not establish which component caused the failure.
IMPORTANT: the GPU UUID at PCI01:00.0 differs from October. October failed GPU-a2ba686c-795e-7b97-53bb-4422a4e8e6f7; this run failed GPU-ba88f0c1-a7d8-0769-c5da-f12f092a4ade. The same PCI address is not evidence that the same physical GPU failed. Do not infer a physical connector assignment from PCI address alone.

## Expected behavior
Calibration completes, or the workload fails without leaving a GPU inaccessible through the driver. Requesting investigation/advice; no claim that this proves a Strata, NVIDIA-driver, PCIe wiring or power-supply root cause.


## Logs and exact configuration
The following sections contain the retained configuration and log text, with local home paths replaced by role placeholders. `<HANDOFF_DIR>` is the local evidence/config workspace; `<STRATA_DATA_DIR>` is the per-user Strata install; `<MODEL_DIR>` is the folder holding all four GGUF shards. These placeholders were not passed literally to Strata.

<details>
<summary>Exact official tuner argv</summary>

```json
[
  "/home/USER/.local/bin/strata",
  "tune",
  "auto",
  "<HANDOFF_DIR>/tune-iq4xs-v31-20261109T103154992378Z/config/base.json",
  "--mode",
  "quick",
  "--time",
  "20",
  "--use",
  "chat",
  "--no-save",
  "--name",
  "iq4xs-v31-44H8-4K",
  "--work-dir",
  "<HANDOFF_DIR>/tune-iq4xs-v31-20261109T103154992378Z/work",
  "--status-file",
  "<HANDOFF_DIR>/tune-iq4xs-v31-20261109T103154992378Z/progress.json",
  "--gpu-lock",
  "<HANDOFF_DIR>/tune-iq4xs-v31-20261109T103154992378Z/gpu.lock",
  "--boot-timeout",
  "600",
  "--output",
  "json"
]
```

</details>

<details>
<summary>Run configuration (all arguments and engine environment overrides)</summary>

```json
{
  "exe": "<STRATA_DATA_DIR>/components/engine/0.1.58-linux-x86_64-cuda-cuda13/strata",
  "cwd": "<HANDOFF_DIR>/tune-iq4xs-v31-20261109T103154992378Z/work",
  "tokenizer": "<HANDOFF_DIR>/config-unsloth-iq4xs-v31-20261109T101945164662Z/pack/tokenizer",
  "args": [
    "--pack",
    "<HANDOFF_DIR>/config-unsloth-iq4xs-v31-20261109T101945164662Z/pack",
    "--native",
    "<MODEL_DIR>/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00004.gguf",
    "--expert-profile",
    "<HANDOFF_DIR>/phase2/support/expert-profile.bin",
    "--expert-cache",
    "auto",
    "--prefill",
    "16384",
    "--spec",
    "4",
    "--spec-min-p",
    "0.5",
    "--mtp",
    "<HANDOFF_DIR>/phase2/support/mtp/rt",
    "--max-context",
    "4096",
    "--kv",
    "int8",
    "--prompt-cache",
    "8",
    "--prompt-cache-every",
    "8192",
    "--kv-resident",
    "0",
    "--vram-reserve-mib",
    "1536",
    "--resident-budget-gib",
    "44",
    "--vram-reserve-later-mib",
    "1100",
    "--pcie-frac",
    "0",
    "--adapt-async",
    "0",
    "--adapt-every",
    "2",
    "--adapt-swaps",
    "96",
    "--pool-workers",
    "7",
    "--pool-tasks",
    "96",
    "--ple-io",
    "direct",
    "--max-new",
    "128",
    "--serve",
    "--gpu",
    "0,1",
    "--layer-split",
    "28",
    "--split-device",
    "1",
    "--suffix-draft",
    "3",
    "--lookup-chain",
    "0",
    "--mtp-max-t",
    "0",
    "--ple-row-cache",
    "1048576",
    "--ple-inflight",
    "256"
  ],
  "env": {
    "STRATA_RESIDENT_HEADROOM_GIB": "8",
    "STRATA_UNBUFFERED_LOAD": "1",
    "STRATA_STAGE_TRIM": "1",
    "STRATA_ROUTE_TAIL_SKIP": "0",
    "STRATA_SPLIT_MTP_BATCH": "1",
    "CUDA_DEVICE_ORDER": "PCI_BUS_ID",
    "CUDA_VISIBLE_DEVICES": "0,1",
    "CUDA_MODULE_LOADING": "EAGER",
    "STRATA_RESIDENT_PIN": "1",
    "STRATA_PIN_GUARD": "2",
    "STRATA_PIN_RESERVE_GIB": "8",
    "STRATA_IO_THREADS": "16",
    "STRATA_SSD_KEEPALIVE": "1",
    "LD_LIBRARY_PATH": "<STRATA_DATA_DIR>/components/engine/0.1.58-linux-x86_64-cuda-cuda13",
    "STRATA_IO_PREFETCH": "0",
    "STRATA_COLD_OVERLAP": "0",
    "STRATA_IO_PF_GATE": "0",
    "STRATA_FILE_TIER_PROBE": "0"
  },
  "gpu": [
    0,
    1
  ],
  "model": "flash-next",
  "name": "unsloth-UD-IQ4_XS-v31-dual5060ti-44GiB-H8-4K"
}
```

</details>

<details>
<summary>Full engine log</summary>

```text
[strata] 2026-11-09 10:32:04 engine started: <STRATA_DATA_DIR>/components/engine/0.1.58-linux-x86_64-cuda-cuda13/strata --serve --pack <HANDOFF_DIR>/config-unsloth-iq4xs-v31-20261109T101945164662Z/pack --native <MODEL_DIR>/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00004.gguf --expert-profile <HANDOFF_DIR>/phase2/support/expert-profile.bin --expert-cache auto --prefill 16384 --spec 4 --spec-min-p 0.5 --mtp <HANDOFF_DIR>/phase2/support/mtp/rt --max-context 4096 --kv int8 --prompt-cache 8 --prompt-cache-every 8192 --kv-resident 0 --vram-reserve-mib 1536 --resident-budget-gib 44 --vram-reserve-later-mib 1100 --pcie-frac 0 --adapt-async 0 --adapt-every 2 --adapt-swaps 96 --pool-workers 7 --pool-tasks 96 --ple-io direct --max-new 128 --serve --gpu 0,1 --layer-split 28 --split-device 1 --suffix-draft 3 --lookup-chain 0 --mtp-max-t 0 --ple-row-cache 1048576 --ple-inflight 256
strata: a model with K-quant experts (Unsloth UD-Q4/Q5/Q6_K_XL) reads its prompt on llama.cpp's MMQ kernels (STRATA_MMQ_KQ=0 or STRATA_NV_KQ_DEFAULT=0 turns it off)
strata generate: memory guard: limit 56.0 GiB (source: cgroup), current 0.8 GiB, reclaimable cache credited 0.0 GiB; available 55.2 GiB (MemAvailable 59.4 GiB), floor 2560 MiB
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 28)
strata generate: native pack: <HANDOFF_DIR>/config-unsloth-iq4xs-v31-20261109T101945164662Z/pack experts (largest blob 2.87 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.4 s)
strata generate: layer split: CUDA0 loads the dense weights of layers 0-27 only
strata generate: 886 MiB of weights loaded from <HANDOFF_DIR>/config-unsloth-iq4xs-v31-20261109T101945164662Z/pack in 0.2 s (603 canonical tensors skipped: served natively)
strata generate: 176 native projection matrices, 1451.77 MiB of weights, in 0.8 s
strata generate: PLE direct read probe: 66138 rows/s (fine)
strata generate: the SSD is kept awake while rows are read: one page of the table after 1 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: layer split: CUDA1 loads the dense weights of layers 28-47 only
strata generate: layer split: CUDA1 holds its weights; 13.62 GiB free (its session follows the split search)
strata generate: profile <HANDOFF_DIR>/phase2/support/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: PLE on, table 320001536 rows of <MODEL_DIR>/Qwen3.8-Flash-Next-UD-IQ4_XS-00004-of-00004.gguf
strata generate: layer split: CUDA1 holds its weights, session [28, 48) and the h

En el sitio

Enlaces a install, modelos, releases.