#6601 [Engine] hip-rx9070 (gfx1201): Inkling-2-mini prompts of 64+ tokens fail in prefill with NaN logits; --set chunk=<64 works

open · @dresraz · 3 comentarios · En GitHub

Server & APIAMD / HIPModels & quantsLinux

Descripción

## Summary

With the Inkling 2 Mini HIP engine, any request whose **rendered prompt reaches ~64 tokens** fails during **prefill** (before the first token) with HTTP 503. At 63 tokens it succeeds; at 64 it fails. `max_tokens=1` fails too, confirming prefill rather than decode. Tool calls fail only because their schema pushes the prompt over the same boundary. A verified launch-time workaround (`--set chunk=<64`) makes all tested prompts and tool calls succeed.

## Environment

| Component | Version |
|---|---|
| Strata | 0.2.19 (build 0.1.65, Python 3.12.15) |
| Engine | `0.1.65-linux-x86_64-hip-rx9070` (`strata-inkling` 0.1.65) |
| GPU | AMD Radeon RX 9070, gfx1201, 16 GB |
| OS | Fedora 45 |
| Models | Inkling-2-Mini SQ-10 (+ vision pack) and SQ-15 (both affected) |

Note: `components engine` first prints "No engine for this PC was found ..." and then resolves the inkling2 engine at 0.1.65; the first line is misleading.

## Minimal repro

Launch the baseline server:

```sh
strata-run launch --model sq-10 --family inkling-2-mini --data-dir <models-dir> \
  --context 64k --vision on --access local --port 8080 --no-tui --yes
```

Plain text, no tools, rendered to >=64 tokens (fails):

```sh
curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"inkling-2-mini",
       "messages":[{"role":"user","content":"<plain text rendering to >=64 tokens>"}],
       "max_tokens":1}'
```

A short control ("Reply with exactly the word: pong", max_tokens 128) returns 200. The same failing result occurs with `max_tokens:1`, with no tools involved, and for tool requests whose non-empty schema crosses 64 tokens.

## Log lines (every failing request)

```
[strata] the engine's logits became non-finite before the first token (#879); it dropped its cached state: sending the request again once
[strata] the engine's logits became non-finite again on the retry (#879): the request ends with an error
[strata] done: 0 tokens in 0 s (no speed) (error, cancel=False)
[strata] 503: the model's output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state - send the request again
```

## Boundaries

Rendered-prompt length (identical NaN on both sides of the boundary):

| Rendered prompt | Result |
|---|---|
| 63 tokens | 200 |
| 64 tokens | 503 (non-finite) |

`--set chunk` (baseline launch + flag; 200/2000/7657-token prompts and tool calls, each run twice after a full stop/relaunch):

| chunk | Result |
|---|---|
| baseline (unset), 64, 128, 256 | 503 / 503 / 503 |
| 16, 32, 40, 48, 56, 60, 62, 63 | 200 / 200 / 200 |

The transition is sharp at 64, suggesting a prefill kernel/tile assumption at chunk >= 64 on gfx1201. All 104 recorded responses were validated: every 200 tool reply contained a real `tool_calls` payload with `{"city":"Paris"}`, and every 503 contained the non-finite message.

## Workaround (verified)

Add `--set chunk=32` (or any value 16-63; 32 gives margin):

```sh
strata-run launch --model sq-10 --family inkling-2-mini --data-dir <models-dir> \
  --context 64k --vision on --access local --port 8080 --no-tui --yes --set chunk=32
```

This makes 200-, 2000- and 7657-token prompts and tool calls with non-empty schema succeed. Cost: the 7657-token prompt took ~28 s at chunk=32. Small PNG images worked in all runs (with and without the flag).

## What did not help

- `--vision cpu`, `--kv fp16`, `--context 8k`, `--set drafting=off`, `--set parallel=1`, `--set batch=1`.
- `--set draft=off` is silently ignored by the launcher (no effect).
- Env knobs `STRATA_HIPBLASLT=0`, `STRATA_HIP_MOE_I8=0`, `STRATA_UMA_COHERENT=1`.
- The names `STRATA_DBG_NAN`, `F16_RANGE`, `PF_FUSED`, `HIP_WMMA`, `HC_Q8`, `SCORES_THREAD` are not present in the 0.1.65 engine binary.
- Switching to the SQ-15 model pack.
- Rejected by the engine (launch exit 6): `--tc`, `--spec`, `--kv-grow`.

## Possibly related

- #6580: gfx1200 illegal memory access in batched prefill at 24+ tokens — possibly the same family of RDNA4 prefill issues; not asserting they are identical.
- #879: older "all-logits-NaN" umbrella, reportedly on a different engine.

En el sitio

Enlaces a install, modelos, releases.