Pull requests / #1681

#1681 bench: community report, RTX 5080 16 GB + RTX 4060 Ti 8 GB (helper cache), IQ3_S at 4K / 32K / 128K on 0.1.41

open · @KarlGrier · 0 commentaires · Sur GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Description

## Title
Issue: none (a results-only report)

## Summary
A results-only report per docs/COMMUNITY_BENCHMARKS.md, measured 2026-10-09. Stock v0.1.41 (`fb58e0db`, source build), ISTA-DASLab IQ3_S at a 131,072-token context, the RTX 5080 as the main GPU and the RTX 4060 Ti as a helper expert cache (`--expert-cache-device1 3300`). Ryzen 9 9950X3D, 96 GB DDR5, PCIe 5.0 NVMe, Linux.

| prompt | prompt tok/s, median (range) | decode tok/s, median (range) | TTFT s, median |
|---|---|---|---|
| 4,096 | 3,451 (3,436-3,500) | 146.0 (135.8-147.3) | 1.20 |
| 32,768 | 4,277 (4,202-4,300) | 132.2 (127.9-137.8) | 7.70 |
| 128,000 | 4,166 (4,165-4,166) | 124.9 (123.7-126.7) | 30.80 |

Three runs per size, each a fresh prompt (0 reused tokens), temperature 0, 256-token cap; `tools/needle_bench.py` 6 of 6 at 32k and 128k.

The README also lists, under their own headings and labelled as not v0.1.41 as released: the same runs on a private build (v0.1.41 + local patches), and two larger files measured on 2026-10-07/08 on an older private build as stand-ins for a bigger model.

## What changed
One new folder, `bench/results/2026-10-09-community-rtx-5080-4060ti-0.1.41/`:
- `README.md`: hardware, software, model, settings, method, results, limitations
- `make_prompts.py`: rebuilds the prompts from the repository's own text at v0.1.41 (sha256 of its output in the script)
- `benchmark.py`: the client (streamed requests, the server's `timings`, memory after each request)
- `results-stock.json`, `results-private.json`, `needles-stock.json`, `needles-private.json`, `summary.json`: per-run data
- `config-stock.json`, `engine-start-stock.txt`: the server config and the engine's start lines, without paths or GPU ids

No code, docs or index files are touched.

## Extra Notes
Not tested: vision, other quantizations, Windows, setup's own configuration for this PC, contexts above 131,072, `--batch`, a layer split, and the 5080 without the helper card.

Measured and drafted with Claude Code.

Sur le site

Liens install, modèles, releases.