#2126 `--batch` hangs during startup on Linux/CUDA (3x RTX 3060, 0.1.43): 100% kernel time, GPU idle, the expert cache is never allocated

说明

## Summary

Adding `--batch` (`"parallel": 2` or `4`) to a working config makes `strata serve` hang during startup, **before the expert cache is allocated**. Without `--batch` the exact same config loads in ~6 minutes and serves normally.

The hang is not I/O, not swap and not a plain RAM shortage: the host spins at **100% system time** with the **GPU completely idle (0%)**, and the kernel reports ~148k failed memory compactions. `strace` cannot attach (container), but `/proc/<pid>/stat` shows `utime` frozen while `stime` advances 1000 ticks per 10 s.

## Environment

| | |
|---|---|
| GPU | 3x NVIDIA GeForce RTX 3060 12 GB (sm_86), driver **580.126.20** |
| PCIe | CUDA0 x4 Gen4, CUDA1/CUDA2 x8 Gen4 (measured) |
| CPU | AMD Ryzen 7 3700X (8c/16t), no AVX-512 |
| RAM | 62 GiB (2x 32 GB DDR4 @2133), swap 16 GiB |
| Motherboard | CVN X570 GAMING PRO |
| OS | Ubuntu 24.04.4 LTS, kernel 6.8.0-142-generic |
| Container | Docker, base `nvidia/cuda:13.0.0-devel-ubuntu24.04`, `--ulimit memlock=-1:-1`, `--shm-size 64m` |
| Engine | **0.1.43**, built from tag `v0.1.43` (`src=e568e70371a3d2e0`), `CMAKE_CUDA_ARCHITECTURES=86`, nvcc 13.0.48 |
| Model | Qwen3.8-Flash-Next GSQ-RCO abliterated **IQ3_S**, native pack `iq3_s_abl`, **48 blocks** |

## Config (the `args` array, identical in every run below unless noted)

```
--pack /data/packs/iq3_s_abl
--native .../Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-00001-of-00002.gguf
--ple-gguf .../Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-00002-of-00002.gguf
--expert-profile /opt/strata/data/expert-profile.bin
--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp /data/mtp/rt
--max-context 235520 --kv int8 --kv-resident 32768 --remote-expert-opt
--control-vector-scaled .../Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf:1.0
--control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --vision
--vram-reserve-mib 700 --trim-stage-weights
```

plus `"gpu": [0,1,2]`, `"layer_split": "16,32"`, and the variable under test `"parallel": 2`.

## Symptom

The log stops at exactly the same line every time, and **never reaches the expert cache allocation**:

```
strata generate: --batch-groups auto: 2 groups of 1 slot
strata generate: --batch: 2 slot sessions on CUDA0 (0.34 GiB each); 9.12 GiB free
strata generate: --batch: 2 slot sessions on CUDA1 (0.34 GiB each); 9.06 GiB free
strata generate: --batch: 2 slot sessions on CUDA2 (0.34 GiB each); 8.51 GiB free
strata generate: --batch 2: the slot sessions take 0.69 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.83 s (952 MiB/s)
strata generate: GPU 0: NVIDIA GeForce RTX 3060, compute capability 8.6
<<< hangs here forever >>>
```

`grep -c "expert cache auto"` on that log is **0**. The next line in a successful (no `--batch`) start is:

```
strata generate: expert cache auto: 5.32 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 1870 slots
```

Measured while hung:

- `nvidia-smi`: **0% utilisation on all three cards**, VRAM frozen at 2578 / 2638 / 4034 MiB (a successful start reaches 11450 / 11354 / 11574 MiB)
- `free`: 6.9 GiB available, swap usage **unchanged** (1437 MiB) — so it is not swapping
- Main thread: `State: R (running)`, `wchan=0`, `/proc/<pid>/syscall` = `running` on 20/20 samples
- `/proc/<pid>/stat`: `utime` frozen (398), `stime` +1001 ticks per 10 s → **100% system time**
- Worker threads: `futex_wait_queue` / `do_poll` (normal)

Kernel counters at the moment of the hang (`/proc/vmstat`):

```
allocstall_movable    608585
allocstall_normal      76862
compact_stall         182850
compact_fail          148075     <-- failed compaction
pgmigrate_success   77873701
pgsteal_direct      68258939
```

`/proc/meminfo`: `AnonPages 46953728 kB`, `Shmem 9353772 kB`, `Mlocked 27356 kB`, `MemAvailable 6089800 kB`.

So the host is repeatedly failing to obtain **contiguous** memory while the GPU sits idle.

## What I ruled out (controlled, one variable at a time)

| Variable | Change | Result |
|---|---|---|
| swap | 4 GiB to **16 GiB** (new swapfile, verified with `swapon --show`) | identical hang; swap usage never grew |
| VRAM reserve | `--vram-reserve-mib 700` to **2000** | identical hang, same log line |
| batch slots | `--batch 2` and `--batch 4` | both hang |
| layer split | **3 stages** (`16,32`, 3 GPUs) and **2 stages** (`20`, 2 GPUs) | both hang |

`--batch` off: all four configs load and serve fine (this is the config I run in production, 13 474 expert slots, hit rate 0.996-0.998, 50-57 tok/s decode).

## Possibly related

- **#2065** - "Engine hangs at layer 48 during decode when VRAM headroom drops below ~500 MiB", `--batch 2`. My working start-up reports exactly `460 MiB of VRAM free with everything loaded`, i.e. below that 500 MiB threshold. My hang is at **start-up** rather than during decode, but the VRAM-headroom figure matches.
- **#2049** - deterministic deadlock "after the last block: host at layer step 48"; this model has 48 blocks too, and the reporter saw it on **both 0.1.41 and 0.1.43**.
- **#1906** / **#1994** - other `--batch` / `--batch-groups` problems still open.
- `docs/BATCHING.md` says *"Only a layer split of two stages on two GPUs has been run"* - so a 3-stage split with `--batch` may be untested territory.

## Notes

- `strace -p` from the host fails with `PTRACE_SEIZE: Operation not permitted` (container isolation), so I could not capture the syscall. If there is a recommended way to trace inside the container, I can rerun and attach the output.
- Happy to run any build with extra instrumentation (`--stats`, debug env vars, a patched binary) on this machine - it reproduces 100% of the time and is otherwise idle.