#2068 `--batch` (concurrent requests) silently kills the engine on 2× Tesla V100-SXM2-16GB (sm_70, x86_64): exit is a signal, no error line

说明

> Report from hardware the maintainers do not have (per `AGENTS.md`). Everything below was measured on one machine.
> The **single-request** path is stable and the numbers are good; only `--batch`/`"parallel": N >= 2` is affected.

## Summary

On an x86_64 host with two V100-SXM2-16GB (sm_70), a layer split across both cards, a native IQ pack and MTP
speculative decoding, the engine **dies as soon as two requests decode together**. Every in-flight request gets
HTTP 503 from the server (`the engine stopped unexpectedly (exit code None)`). The engine process is gone; the
server restarts it lazily on the next request. Single requests (one at a time) work.

- `--batch 2` crashes. `--batch 4` crashes.
- Reproduced **without** `--kv-resident` and at `--max-context 32768`, so it is not KV streaming or a long context.
- The engine requires `--spec >= 2` for a native IQ pack, so `--spec`/MTP could not be removed to test that alone.
- No CUDA error, no "out of memory", no assert is printed: the process is killed by a signal (`exit code None`).

## Environment

| | |
| --- | --- |
| Fork / branch | `eelgaev/Strata-AC922`, branch `ac922` |
| Commit | `1f029aadc0441ebc07ab00f411f2537359ceace3` (2026-10-09, "docs: merged up to upstream v0.1.41") |
| Engine version (from the log) | `session is up (engine 0.1.41)` |
| Host CPU | 2× Intel Xeon E5-2686 v4 (2 sockets, 18 cores each, AVX2, no AVX-512), 94 GiB RAM (2 NUMA nodes × 48 GB) |
| Host arch | x86_64 |
| OS / toolchain | Ubuntu 24.04, glibc 2.39, gcc/g++ 13.3, CUDA 12.9 (`/usr/local/cuda-12.9`) |
| Driver | 580.178.04 |
| GPUs | 2× Tesla V100-SXM2-16GB (sm_70, no display), nvidia-smi index 2 and 3 |
| GPU link | PCIe 3.0 **x8** per card (the board's root ports are x8), NVLink NV6 between the two; P2P works both ways |

Build (x86_64, the engine is compiled locally):

```sh
cmake -S . -B build-cuda12 -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \
  -DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=70 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.9/bin/nvcc \
  -DSTRATA_GGML_DIR=third_party/llama.cpp
cmake --build build-cuda12 --target strata -j
```

(`third_party/llama.cpp` is the pinned commit `3cf03257f219afbe7334045ff7c6a06ac68c627d`.)

## Model and config

- Qwen3.8-Flash-Next **GSQ-RCO IQ2_XS**, two shards, as a **native (IQ) pack** (`tools/iq_pack.py`).
- MTP draft layer from `Qwen/Qwen3.8-Flash-Next` (`tools/mtp_fetch.py` → `mtp_pack.py` → `mtp_rt.py`).
- Config (`strata-iq2_xs.json`), layer split `auto` resolved to K=24 (CUDA0 layers 0–24, CUDA1 25–47):

```json
{
  "exe": ".../engine-cuda12/strata",
  "parallel": 4,
  "args": [
    "--pack", ".../packs/iq2_xs",
    "--native", ".../Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf",
    "--ple-gguf", ".../Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf",
    "--expert-profile", ".../data/expert-profile.bin",
    "--expert-cache", "auto", "--prefill", "auto",
    "--spec", "4", "--spec-min-p", "0.5", "--mtp", ".../mtp/rt",
    "--max-context", "262144", "--kv", "int8", "--kv-resident", "32768"
  ],
  "gpu": [2, 3], "layer_split": "auto"
}
```

Startup lines (both cards fit their sessions):

```
strata generate: layer split auto: K=24 - predicted 36.5 ms per decode window; the caches hold 15971 of 24576 profiled pairs (~98.8% of the routed mass), best of 46 placements
strata generate: KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM
strata generate: --batch-groups auto: 2 groups of 2 slots
strata generate: --batch: 4 slot sessions on CUDA0 (0.51 GiB each); 9.85 GiB free
strata generate: --batch: 4 slot sessions on CUDA1 (0.51 GiB each); 9.52 GiB free
strata generate: --batch 4: the slot sessions take 2.04 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata generate: session is up (engine 0.1.41)
strata verify: batch windows of up to 4 sequences (layers [0, 24))
strata verify: batch windows of up to 4 sequences (layers [24, 48))
```

## Steps to reproduce

1. Install IQ2_XS as a native pack and start the server with the config above (one request at a time works).
2. Send **two** chat completions at the same time:

```sh
for i in 1 2; do
  curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d "{\"model\":\"x\",\"messages\":[{\"role\":\"user\",\"content\":\"Write ~120 characters about season $i.\"}],\"max_tokens\":160,\"temperature\":0,\"reasoning_effort\":\"none\"}" &
done
wait
```

3. Both responses are `503`:

```json
{"error": {"type": "server_error", "message": "the engine stopped unexpectedly (exit code None); the next request restarts it"}}
{"error": {"type": "server_error", "message": "the engine stopped while this request waited; it was not sent; the next request restarts it"}}
```

The engine process no longer exists (`pgrep` finds none); the next request makes the server start it again.

## Engine log around the crash

The log simply stops after the first batched verify window and a fresh startup follows. There is **no** error line
between them:

```
...
strata serve: conversation cache: parked 16 tokens in 360.9 ms; parked=2 bytes=474292728 ...
strata serve: prompt 29 tokens = 0 reused + 29 read in 783 ms (37.0 tok/s), 1 generated in 25 ms (39.6 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 74.9% (355 hits / 474 lookups)
strata serve: KV streaming: 93.33% of 720 block reads hit VRAM, 0.2 MiB read from RAM
strata batch: slot 0 takes 30 tokens (copied in 367.4 ms)
strata serve: prompt 30 tokens = 29 reused + 1 read in 0 ms (10401.5 tok/s), 1 generated in 23 ms (44.3 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata verify: captured the batch window over slots 0
[strata] 2026-10-12 15:15:09 engine started: ...          <-- restart, new process; the old one vanished
```

Free VRAM in the verify windows was low but not zero (396–584 MiB reported by the `captured the ... window` lines).
No `cudaError`, no `CUDA error`, no `out of memory`, no assertion reaches the log.

## What I ruled out

| Variation | Result |
| --- | --- |
| `parallel: 1` (one at a time), same config | stable; many requests in a row |
| `parallel: 2` → `--batch 2` | **crash** (both requests 503) |
| `parallel: 4` → `--batch 4` | **crash** |
| same, but **no** `--kv-resident` and `--max-context 32768` | **crash** (so it is not KV streaming / 256K) |
| any of the above with a **single** request | works |

The native pack requires `--spec >= 2` (the engine refuses without it: *"it needs --native SHARD1, --spec T (T >= 2)
and --prefill CHUNK"*), so I could not test whether `--spec`/MTP is required for the crash.

## Notes / hypotheses

- The batched verify path is the prime suspect: the last line before the death is the first
  `captured the batch window over slots 0`, and nothing after it is printed.
- The layer split's verify path was heavily changed on this fork (peer fetch, async commit); sm_70 may have a path
  that is not exercised on the maintainers' machines.
- Only ~400–580 MiB of VRAM were free in the earlier windows, and a batched window needs more buffers per slot; an
  allocation failing without a printed error would fit, but an OOM usually prints something. The silent signal exit
  looks more like a crash.
- `--batch-mtp` (per-slot MTP) was **not** enabled; this is the plain `--batch N` path with `--spec 4`.

## Aside: x86_64 build needs `kq_vsx.cpp` compiled (small, separate)

On x86_64 the link fails with `undefined reference to strata::kernels::cpu::kq_vsx_on()` /
`kq_vsx_gu_rows()` / `kq_vsx_q51_rows()`, because `native_expert.cpp` references them on every CPU, while
`CMakeLists.txt` only adds `src/kernels/cpu/kq_vsx.cpp` when `CMAKE_SYSTEM_PROCESSOR MATCHES "^(ppc64|powerpc64)"`.
`kq_vsx.cpp` already has an `#else` branch with the stubs (`kq_vsx_on()` returns false, the row functions `abort()`),
so compiling it everywhere fixes the link and changes nothing on POWER. Local patch:

```cmake
target_sources(strata_kernels_cpu PRIVATE src/kernels/cpu/native_expert.cpp src/kernels/cpu/kq_vsx.cpp)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(ppc64|powerpc64)")
  set_source_files_properties(src/kernels/cpu/kq_vsx.cpp PROPERTIES COMPILE_OPTIONS "-mcpu=power9")
endif()
```

Happy to test a patch or a debug build (e.g. one that turns CUDA errors into a printed message, or `cuda-memcheck`)
on this machine.