Issues / #3124
#3124 Request: benchmark config behind 0.1.45 UD-Q4_K_XL decode 117.8 tok/s (R9700)
open · @vvsvvt · 0 comentarios · En GitHub
BenchmarksSetup & installAMD / HIPModels & quantsWindows
Descripción
Hi, the 0.1.45 release notes say: > R9700 32 GB, UD-Q4_K_XL: decode 83.2 to 117.8 tok/s (1.42x, 5 of 5 pairs) I've been trying to reproduce the 117.8 number on the same hardware (R9700 32GB, Windows, gfx1201) running 0.1.45 with UD-Q4_K_XL (Unsloth), and I'm consistently well below it. Before digging further, could you share the exact benchmark setup behind that number? My best results, ~4K prompt, single request, 3 runs each (first discarded as warmup), decode tok/s from the server log: | config | best decode | |---|---| | --spec 6, --pcie-frac 0.2, --kv int8, --pool-workers 5, --prefill 8192, --resident-budget-gib 60 | 79.5 | | --spec 2 (same otherwise) | 68.5 | | --pcie-frac 0, --spec 6 (same otherwise) | 74.9 | Full args for the 79.5 run: --pack strata-data/packs/ud-q4_k_xl --native <UD-Q4_K_XL gguf> --resident-budget-gib 60 --vram-reserve-mib 1024 --expert-profile strata/data/expert-profile.bin --expert-cache auto --pcie-frac 0.2 --pool-workers 5 --prefill 8192 --spec 6 --spec-min-p 0.7 --mtp strata-data/mtp/rt --kv int8 --kv-resident 32768 --max-context 262144 env: STRATA_HIP_WMMA=1, STRATA_SELECT_WMMA=1, STRATA_MMQ_KQ=1, STRATA_ROUTE_RESIDENT=0.25 What I'd like to know for the 117.8 run: 1. Full command line / config (especially --spec/--mtp, --pcie-frac, --kv, --resident-budget-gib, --pool-workers) 2. Prompt length and generated tokens per run 3. How decode tok/s was measured (server log "done" line? averaged over how many runs?) This would help me (and other R9700 users) tell whether the gap is config or something else. Thanks!
En el sitio
Enlaces a install, modelos, releases.