#6580 [Engine] 0.1.65 HIP gfx1200: illegal memory access in batched prefill (Inkling-2 GGUF, NVFP4→SF4 experts); short prompts are fine

open · @oakd2003 · 0 comentários · No GitHub

BenchmarksAMD / HIPModels & quants

Descrição

### Summary
With the GGUF loading fix from #6579 applied, Inkling-2-Mini GGUF runs on gfx1200
for short prompts, but any longer prompt aborts with a HIP illegal memory access
during prefill.

### Environment
Same as #6579: v0.2.19 / engine 0.1.65 / gfx1200 / ROCm 7.10.0a20251120,
`Inkling-2-Mini-JP-16GB-UD-Q3_K_XL.gguf` (loop model: 24 blocks, `num_loops=3` → 48
visits, SWA window 1024 on 40 of 48 layers, experts NVFP4 → SF4 `code_id=15`).

### Reproduction
```
# OK
strata-inkling --bench --native model.gguf --gpus 1 --prompt-len 8  --gen 2
[bench] backend=hip format=gguf parallel=1 gpus=1 kv=fp16 prompt=8 gen=2 | prefill=183.7 ppts (0.04s) | decode=39.0 tps agg

# FAILS at 24 tokens: "logits D2H: an illegal memory access was encountered"
strata-inkling --bench --native model.gguf --gpus 1 --prompt-len 24 --gen 4

# FAILS at 32 tokens: "tdr visit flush: an illegal memory access was encountered"
strata-inkling --bench --native model.gguf --gpus 1 --prompt-len 32 --gen 8
bench prefill err: [ink-hip] tdr visit flush: an illegal memory access was encountered
Hip error: 'an illegal memory access was encountered'(700) at .../hipblaslt/library/src/amd_detail/hipblaslt.cpp:170

# FAILS
strata-inkling --bench --native model.gguf --gpus 1 --prompt-len 128 --gen 32
```

### What is already ruled out
- Not the loader: the same file runs on the CPU backend (with the fix applied).
- Not the manifest/shapes: 8-token prefill + decode on the HIP backend is coherent.
- Measured threshold: 8 and 16 prompt tokens pass; 24 and 32 fail. At 24 the failure
  surfaces at `logits D2H`, at 32 at `tdr visit flush` in `HipBackend::forward_batch`
  (a sticky error then shows up from hipBLASLt, so that line is probably a bystander).

### Next steps I can help with
Happy to test the Q4_K_XL (non-NVFP4 expert) variant of the same model to separate the
NVFP4→SF4 repack path from the generic batched-prefill path, or to run with
`STRATA_HIP_PROF=1` / `STRATA_HIP_TREE_PROF=1` if those print anything useful on 0.1.65.

No site

Links install, modelos, releases.