Pull requests / #2540

#2540 bench: Coder IQ1_M on RTX 3060 12 GB, i7-7700K (4 cores), Linux, 128K context

open · @fszlin · 0 comentarios · En GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsLinux

Descripción

Results-only community report (no engine changes).

Hardware: RTX 3060 12 GB (PCIe 3.0 x16), i7-7700K (4C/8T, AVX2, no AVX-512),
46.9 GiB RAM, Samsung 970 EVO Plus NVMe (PCIe 3.0 x4) for the model files,
Debian 13 (Linux 6.12), driver 615.71.09, CUDA 13.1 toolkit.
Software: Strata 0.1.44.1 (commit 296b59d2), setup's own configuration with
--context 131072 and --draft-vocab en.
Model: Qwen3.8-Flash-Next Coder, IQ1_M; all 23.4 GiB of experts in RAM,
2,666 experts (5.11 GiB) in the GPU cache.

`tools/strata_bench.py --long`, 3 runs per test, thinking off, medians:
- chat decode 34.3 tok/s (30.1-41.0), first token 1.10 s
- fresh prompts: 4K 974 tok/s, 16K 1,111 tok/s, 32K 1,110 tok/s (32K first token 28.9 s)
- agent session: cache hit 68.6%, turn first token 3.5 s, decode 36.6 tok/s

A same-machine run at 32K context gave the same results within noise (agent
decode about 15% higher at 32K, partly run-to-run variation).

Not tested: answer quality, recall, a real 128K prompt, thinking mode,
power or clock capture. Other Docker services were running during the run.

En el sitio

Enlaces a install, modelos, releases.