Issues / #2871
#2871 Windows HIP: limit the #1630 prefill cap and warning to the affected archs (gfx1030 is stable at 8192+)
open · @ByteJuggler · 0 comentarios · En GitHub
BenchmarksSetup & installAMD / HIPModels & quantsWindows
Descripción
Follow-up to #1630 (closed in the 3-day clean-up): another data point, on a different AMD card: **On an RX 6900 XT (gfx1030) on Windows 11, chunks of 8192 tokens and above have been completely stable, never slower, and up to 12% faster on long prompts.** So the crash in #1630 may be specific to gfx1100 or its prompt path, not to Windows HIP or AMD Radeon as a whole. **Setup** - Engine 0.1.44.1, ready-made HIP zip (ROCm 10.2.0a20260930). - RX 6900 XT 16 GB, gfx1030, AMD Software 26.8.1. *gfx1030 has no hipBLASLt table, so the plain hipBLAS path runs.* - Windows 11 Pro 23H2. - i7-5930K, 128 GB DDR4-2133 (64 GB and 92 GB for some earlier runs), 32 GB page file. - Qwen3.8-Flash-Next Q2_0, IQ3_S and IQ3_XXS. - Flags: `--kv int8 --kv-resident 20480 --vram-reserve-mib 1024 --pool-tasks 36`, `STRATA_HIP_PROMPT_F16=1`, `--max-context` 131072 up to 262144. **Stability** - 45 engine starts in tuning sweeps with `--prefill` at 8192, 10240, 12288 or 16384, plus the servers started by hand for the long-context tests. - Prompts from 4K up to 255K tokens, with 8192 and 12288 as the usual settings. - No crashes and no 0xC0000409. - Commit charge stayed flat. During a run of 64K–248K prompts at `--prefill 12288` and 262144 context (IQ3_XXS), the system commit peaked at 75.4 GB and the engine's working set at 45.3 GB, sampled every 30 s. **Speed** (prompt tok/s, Q2_0 at 131072 context, compared with 8192; decode unchanged) | `--prefill` | 4K | 16K | 32K | 60K | |---|---|---|---|---| | 4096 | −18% | −18% | −19% | −20% | | 7168 | 0 | −12% | 0 | −2% | | 10240 | 0 | +7% | −1% | +2% | | 12288 | 0 | +3% | +11% | +5% | - IQ3_S, 150K prompt: 895 tok/s at the default `auto` (capped at 6144), 1,007 tok/s at `--prefill 12288` (+12%). - At 262144 context with 12288 chunks, 255K prompts read at 950–990 tok/s. **Suggestion** On gfx1030, *the `auto` cap of 6144 and the warning at every start cost about 10% on long prompts*, for a crash that hasn't shown up on this card. **So,** could the cap and the warning apply only to the archs where it was seen (gfx1100, or the hipBLASLt prompt path if that's where it comes from)? I can run a build with the cap lifted for gfx1030 and measure plain `--prefill auto`, if that would help.
En el sitio
Enlaces a install, modelos, releases.