Issues / #2313

#2313 [Bug]: --calibrate on Linux/HIP (RX 7800 XT eGPU over Thunderbolt): engine stops unexpectedly (exit code -15) with no ERR line

open · @RandallSP · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

**Environment**

- Strata 0.1.43.1, engine 0.1.43.1, commit `a0116b27` (which was `origin/main` at the time).
- HIP backend on an RX 7800 XT (gfx1101, eGPU over Thunderbolt) beside an RTX A2000; CachyOS Linux, kernel 7.2.9-2-cachyos; 124 GB RAM.

**What happened**

`--calibrate` aborts with the engine gone (exit code -15), on a Linux + HIP setup.

1. `./setup.sh --calibrate` (config `strata-iq3_s.json`, engine 0.1.43.1, commit a0116b27, which was `origin/main` at the time).
2. The PCIe-share sweep, the draft-floor sweep, and the 7- and 5-worker stages all completed and printed their medians.
3. During the 4-worker stage the engine started (its log banner: `2026-10-13 10:57:48 engine started`), reached `session is up`, and served requests.
4. At about 11:00:5x the engine died. setup printed:

```
  [!]  the tuning did not finish (the engine stopped unexpectedly (exit code -15)): the default settings stay
       the engine said: strata serve: suffix drafts: 1 windows, 1 of 1 drafts accepted
Exception ignored while finalizing file <_io.TextIOWrapper name=8 encoding='utf-8'>:
Traceback (most recent call last):
  File "/mnt/NX/Strata/setup.py", line 4204, in calibrate_config
    return False
BrokenPipeError: [Errno 32] Broken pipe

  [!]  this PC is NOT tuned: the tuning failed (the reason is above); the model starts with the default settings
```

Expected: the tuning finishes, or a start that fails inside a later stage drops that candidate (#1337) instead of ending the whole run.

**What I checked** (no supervisor here, no llama-swap; the engine was started by setup itself)

- Engine log: no `ERR verify` line that day and no `issue #29` line (the only one in the file is from 2026-10-11).
- Kernel: `ring .* timeout` 0, `device wedged` 0, `reset compute queue` 0, `amdgpu_pmops_suspend` 0, no devcoredump entry. The eGPU stayed present (`runtime_status=active`, link 16 GT/s x16).
- VRAM came back within a minute: `mem_info_vram_used` 16,959,606,784 B -> 191,897,600 B.
- Code reading: `-15` is a SIGTERM. `setup.py` and `tools/calibrate.py` cannot send one: their stop path is `QUIT` then `proc.kill()` (SIGKILL, -9), and the READY timeout (`ENGINE_READY_S`, 900 s) also kills with SIGKILL and reports a different message. Only `serve/server.py` sends SIGTERM to the engine: `_pump` on a `FATAL_PREFIXES` line, and `close()` when `QUIT` does not end the engine in 20 s. Neither trace is in the log.
- So the sender of the SIGTERM is unknown from here. The message text matches #2257, where the SIGTERM came from outside (a supervisor signalling the whole process group).

**Engine log tail** (`strata-iq3_s.log`, the 4-worker engine, last 20 lines)

```
strata generate: session is up (engine 0.1.43.1)
strata generate: token graph hit path: 4838 resident experts, decided on the device
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata serve: the prompt path borrows 2210 CUDA0 cache slots (4.18 GiB)
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 74.7 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: adaptive overlap: off (STRATA_ADAPT_OVERLAP=1 turns it on)
strata serve: 614 MiB of VRAM free with everything loaded
strata verify: capturing the 4-token window (519 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (489 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (487 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (487 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata serve: prompt 774 tokens = 0 reused + 774 read in 16216 ms (47.7 tok/s), 111 generated in 5090 ms (21.8 tok/s), drafts accepted 57 of 110, 1 checkpoints
strata serve: decode expert cache hit rate: 61.5% (48678 hits / 79158 lookups); 522 more read by the GPU over PCIe or from another GPU (0.7% of all 79680 routed)
strata serve: KV streaming: 99.37% of 424044 block reads hit VRAM, 10.7 MiB read from RAM
```

The log ends there (file mtime 11:05:12). No error line, no QUIT line, nothing after it.

**Config**: `strata-iq3_s.json` — IQ3_S pack, `--vision cpu`, `--kv int8`, `--kv-resident 32768`, `--max-context 65536`, `--vram-reserve-mib 700`, `--prompt-cache-every 4096`.

Nothing was killed by hand, no OOM line in the kernel log, and the machine kept running the whole time.

Sur le site

Liens install, modèles, releases.