Pull requests / #1987

#1987 Community benchmark: RTX 4090 D 48 GB + Flash-Next IQ3_XXS (0.1.43), 4K/16K/32K fresh 2,314/3,333/3,528 tok/s

open · @jy02375437-max · 0 commentaires · Sur GitHub

BenchmarksNVIDIA / CUDAModels & quantsDocumentationWindows

Description

Adds a community benchmark report for a single-GPU configuration that is not in the list yet: RTX 4090 D 48 GB, original Flash-Next GSQ-RCO **IQ3_XXS**, all experts resident, 262,144-token context, engine 0.1.43 (official Windows release, project commit 7695859a).

Measured with `tools/strata_bench.py --url http://127.0.0.1:1234 --long` against an already running server — the tool started nothing, so I added `engine-log-excerpt.txt` by hand with the exact flags (the tool writes that file only when it starts the server itself). `README.md` and `results.json` are the tool's output unchanged; home paths are sanitized and no credentials are included.

Highlights (medians of 3 runs each, speed suite only — no correctness check in the suite):

- fresh prompts 4,096 / 16,384 / 32,768 tokens: **2,314 / 3,333 / 3,528 prompt tok/s**, first token 1.76 / 4.85 / 9.14 s
- short chat: decode 170.2 tok/s, first token 0.24 s
- agent session (5 follow-up turns): conversation cache hit **68.8 %**, turn first token 1.62 s, decode 158.6 tok/s

Two things that may be worth knowing when reading the numbers: the card is on a **PCIe 4.0 x4 link** (the engine's own probe measured about 5.1 GB/s), which is why prompt throughput is where it is; and the configuration pins `--ple-io direct` (the boot probe's automatic `--ple-io mmap` fallback wedges on this box — reported separately as #1984).

Retrieval on this box was checked outside this suite (32K needles 9/9 on the same configuration), but that is not part of this report.

Sur le site

Liens install, modèles, releases.