Issues / #690
#690 Layer split: native prefill kernels fail on the second GPU (no kernel image available), regardless of which card it is
open · @Nauclerus · 2 comments · View on GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Description
## Summary
A two-GPU AMD layer split starts normally, decodes normally, and reads short prompts normally — but the **first prompt larger than the prompt chunk fails** with `no kernel image is available for execution on the device`, thrown by the native prefill kernels.
The failure follows the **second device in the split**, not the architecture: swapping the card order fails identically. Each card works fine on its own with the same binary.
## Machine
- openSUSE Tumbleweed, kernel `7.2.7-1-default`, user in the `render` group
- Ryzen 7 5800X3D (no AVX-512), 62 GiB RAM
- GPU 0: RX 7900 XTX (gfx1100), 24 GB — boot GPU, drives the desktop
- GPU 1: RX 6950 XT (gfx1030), 16 GB
- ROCm 7.10.0a20251120 from setup's own wheels (`rocm_sdk_core` / `rocm_sdk_devel` / `rocm_sdk_libraries_gfx110X-dgpu`), hipBLASLt 1.2.x
- Engine 0.1.38, built by setup with both arches:
```
cmake -G Ninja -S strata -B build-hip -DCMAKE_BUILD_TYPE=Release \
-DSTRATA_ENABLE_HIP=ON -DSTRATA_ENABLE_CUDA=OFF -DSTRATA_BUILD_TESTS=OFF \
-DSTRATA_PREFILL_MMQ=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030;gfx1100 \
-DCMAKE_HIP_COMPILER=.../llvm/bin/clang++ -DCMAKE_HIP_COMPILER_ROCM_ROOT=... -DCMAKE_PREFIX_PATH=...
```
`engine/BUILD.json`: `{"backend":"hip","version":"0.1.38","archs":["gfx1030","gfx1100"],"src":"d844541998f4d168"}`
Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS, engine args `--expert-profile --expert-cache auto --prefill auto --spec 4 --mtp --kv int8 --kv-resident 32768 --max-context 262144 --vram-reserve-mib 5120`.
## Repro
Config `"backend": "hip", "gpu": [0, 1], "layer_split": "auto"`, start the server, then send a prompt bigger than the prompt chunk (8192 tokens here):
```
$ curl -s http://127.0.0.1:8082/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen3.8-flash-next-iq2_xs","messages":[{"role":"user","content":"<44022 tokens>"}],"max_tokens":4}'
{"error": {"type": "invalid_request_error", "message": "prefill PLE: native PLE postops launch: no kernel image is available for execution on the device"}}
```
Startup looks healthy:
```
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: layer split auto: K=34 - predicted 32.4 ms per decode window; the caches hold 19023 of 24576 profiled pairs (~99.3% of the routed mass)
strata serve: layer split: layers 0-33 (CUDA0), 34-47 (CUDA1), one hand-off per window
```
The same request on the same server with a smaller prompt succeeds, so the split itself works:
```
strata serve: prompt 617 tokens = 0 reused + 617 read in 1779 ms (346.7 tok/s), 1273 generated in 16288 ms (78.2 tok/s)
strata serve: prompt 6553 tokens = 0 reused + 6553 read in 10357 ms (632.7 tok/s), 14 generated in 241 ms
prefill swiglu_interleaved: no kernel image is available for execution on the device
```
6553 tokens (one chunk) is the largest prompt that succeeds; the next request — the first one that spans more than one chunk — fails. Both native prefill kernels I hit are the same shape of failure: `prefill swiglu_interleaved` and `prefill PLE: native PLE postops launch`.
## Controls that pass
- **6950 XT alone** (`gpu: 1`), same binary, same 44022-token prompt: HTTP 200.
- **7900 XTX alone** (`gpu: 0`), same binary, 44022-token and 110022-token prompts: HTTP 200.
- So both code objects are in the binary (BUILD.json lists both archs), and the engine's own arch check is happy with gfx1030. An earlier single-arch build correctly refused the card at startup, which shows the check works:
```
strata generate: GPU 1 (AMD Radeon RX 6950 XT, gfx1030) is not an architecture this Strata engine was compiled for (gfx1100); compile it for this card ...
```
- **Order does not matter.** With `gpu: [1, 0]` (6950 first, XTX second) the same error appears, this time on the XTX. `CUDA_MODULE_LOADING=LAZY` does not change it.
## What I think is happening
`native_ple_postops.cu` has no arch guards; the launch is a plain `<<<...,st>>>` followed by `launch_check()` (lines 197–199, the throw at 171), and `prefill.cpp` does set the device per stage (`cudaSetDevice(pp->dev)`), so the kernel is dispatched to the right card — but the runtime has no code object for it there. Since the failure tracks "the second device" rather than "the gfx1030 card", my guess is that the fatbinary's code object is selected once, for the device that is current when the module is first loaded, and a second device with a different arch gets nothing.
Two questions:
1. Is the code object loaded per device, or once against device 0?
2. Is there a way to turn the native prefill kernels off per device? `--native` enables them (`src/program/generate.cpp:1618`) and `--native-ple-postops` is opt-in only — there is no off-switch, so I cannot test whether the reference path would work on the second card.
Side note, probably not the cause but worth mentioning: setup's wheels on this machine are the `gfx110X-dgpu` variant, so there is no gfx1030 Tensile library, and no hipBLASLt tuning table for gfx1030 with this hipBLASLt version. The failing kernels are the engine's own, not hipBLASLt.
Happy to test a patch or a build.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.