Pull requests / #2540
#2540 bench: Coder IQ1_M on RTX 3060 12 GB, i7-7700K (4 cores), Linux, 128K context
open · @fszlin · 0 Kommentare · Auf GitHub
BenchmarksSetup & installNVIDIA / CUDAModels & quantsLinux
Beschreibung
Results-only community report (no engine changes). Hardware: RTX 3060 12 GB (PCIe 3.0 x16), i7-7700K (4C/8T, AVX2, no AVX-512), 46.9 GiB RAM, Samsung 970 EVO Plus NVMe (PCIe 3.0 x4) for the model files, Debian 13 (Linux 6.12), driver 615.71.09, CUDA 13.1 toolkit. Software: Strata 0.1.44.1 (commit 296b59d2), setup's own configuration with --context 131072 and --draft-vocab en. Model: Qwen3.8-Flash-Next Coder, IQ1_M; all 23.4 GiB of experts in RAM, 2,666 experts (5.11 GiB) in the GPU cache. `tools/strata_bench.py --long`, 3 runs per test, thinking off, medians: - chat decode 34.3 tok/s (30.1-41.0), first token 1.10 s - fresh prompts: 4K 974 tok/s, 16K 1,111 tok/s, 32K 1,110 tok/s (32K first token 28.9 s) - agent session: cache hit 68.6%, turn first token 3.5 s, decode 36.6 tok/s A same-machine run at 32K context gave the same results within noise (agent decode about 15% higher at 32K, partly run-to-run variation). Not tested: answer quality, recall, a real 128K prompt, thinking mode, power or clock capture. Other Docker services were running during the run.
Mehr auf der Site
Links zu Install, Modellen, Releases.