Issues / #2871

#2871 Windows HIP: limit the #1630 prefill cap and warning to the affected archs (gfx1030 is stable at 8192+)

open · @ByteJuggler · 0 Kommentare · Auf GitHub

BenchmarksSetup & installAMD / HIPModels & quantsWindows

Beschreibung

Follow-up to #1630 (closed in the 3-day clean-up): another data point, on a different AMD card:

**On an RX 6900 XT (gfx1030) on Windows 11, chunks of 8192 tokens and above have been completely stable, never slower, and up to 12% faster on long prompts.**

So the crash in #1630 may be specific to gfx1100 or its prompt path, not to Windows HIP or AMD Radeon as a whole.

**Setup**
- Engine 0.1.44.1, ready-made HIP zip (ROCm 10.2.0a20260930).
- RX 6900 XT 16 GB, gfx1030, AMD Software 26.8.1. *gfx1030 has no hipBLASLt table, so the plain hipBLAS path runs.*
- Windows 11 Pro 23H2.
- i7-5930K, 128 GB DDR4-2133 (64 GB and 92 GB for some earlier runs), 32 GB page file.
- Qwen3.8-Flash-Next Q2_0, IQ3_S and IQ3_XXS.
- Flags: `--kv int8 --kv-resident 20480 --vram-reserve-mib 1024 --pool-tasks 36`, `STRATA_HIP_PROMPT_F16=1`, `--max-context` 131072 up to 262144.

**Stability**
- 45 engine starts in tuning sweeps with `--prefill` at 8192, 10240, 12288 or 16384, plus the servers started by hand for the long-context tests.
- Prompts from 4K up to 255K tokens, with 8192 and 12288 as the usual settings.
- No crashes and no 0xC0000409.
- Commit charge stayed flat. During a run of 64K–248K prompts at `--prefill 12288` and 262144 context (IQ3_XXS), the system commit peaked at 75.4 GB and the engine's working set at 45.3 GB, sampled every 30 s.

**Speed** (prompt tok/s, Q2_0 at 131072 context, compared with 8192; decode unchanged)

| `--prefill` | 4K | 16K | 32K | 60K |
|---|---|---|---|---|
| 4096 | −18% | −18% | −19% | −20% |
| 7168 | 0 | −12% | 0 | −2% |
| 10240 | 0 | +7% | −1% | +2% |
| 12288 | 0 | +3% | +11% | +5% |

- IQ3_S, 150K prompt: 895 tok/s at the default `auto` (capped at 6144), 1,007 tok/s at `--prefill 12288` (+12%).
- At 262144 context with 12288 chunks, 255K prompts read at 950–990 tok/s.

**Suggestion**

On gfx1030, *the `auto` cap of 6144 and the warning at every start cost about 10% on long prompts*, for a crash that hasn't shown up on this card.

**So,** could the cap and the warning apply only to the archs where it was seen (gfx1100, or the hipBLASLt prompt path if that's where it comes from)?

I can run a build with the cap lifted for gfx1030 and measure plain `--prefill auto`, if that would help.

Mehr auf der Site

Links zu Install, Modellen, Releases.