#6596 [Benchmark] community: RTX 4090 48 GiB, Qwen3.8-Flash-Next unsloth UD-IQ3_XXS, 262k ctx

open · @sd133cn · 0 comments · View on GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationWindows

Description

## Problem and result

Results-only community benchmark: adds `bench/results/2026-11-22-community-rtx-4090-qwen3-8-flash-next-unsloth-ud-iq3-xxs/` (README.md, results.json, engine-log-excerpt.txt). No code changes. It gives other users a datapoint for Qwen3.8-Flash-Next unsloth UD-IQ3_XXS on an RTX 4090 (48 GiB VRAM) measured with the repo's own quick suite.

Results, quick suite 1 (`tools/strata_bench.py`), 3 runs per test, thinking off, temperature 0, the install's own configuration, 262,144-token context. Medians (full ranges in the README):

- short chat: decode 174.5 tok/s (139.3-332.1, n=9), first token 0.23 s
- 4K fresh prompt: 3520.5 tok/s, first token 1.17 s
- 16K fresh prompt: 5012.3 tok/s, first token 3.25 s
- agent session (6 turns, 4 tools, 3 sessions): 69.2% cache hit, 5 re-read tokens per session, turn first token 1.00 s, decode 176.7 tok/s

## Related discussion

- #2530 — same card, UD-IQ4_XS
- #6422 — RTX 5090, same quant (open)

Scope agreement: none needed, a results-only benchmark report can go straight to a PR per CONTRIBUTING.md.

## Approach

- The folder is what the tool writes (`bench/results/<date>-community-<gpu>-<model>/`): README.md in the suggested report format, results.json with every request's tokens and timings, engine-log-excerpt.txt with the server's start-up lines.
- The install's own configuration, unchanged; the exact launch command is in the README. Machine: NVIDIA GeForce RTX 4090, 48 GiB VRAM; i7-14700KF (20C/28T); 95.8 GiB RAM; Samsung 990 PRO NVMe; Windows 11 (10.0.26320); driver 621.24 (CUDA up to 13.5).

## Validation

**Passed:**

- Command: `python tools/strata_bench.py` (suite 1, 3 runs per test, thinking off, temperature 0) against a Strata server started by the benchmark from the install configuration; the README's Method section and results.json hold the exact settings of every request.
- Revision/environment: Strata project tools 0.1.61, engine 0.1.59.1 (commit hash not recorded, see Limits); Windows 11; RTX 4090 48 GiB; driver 621.24.
- Observed: every measured request returned tokens (9 chat + 3×4K + 3×16K + 18 agent turns); suite total 48 s. Medians listed above; per-run numbers in README.md, per-request data in results.json.

**Failed:**

- none observed in this run. (An earlier attempt on 2026-11-08 on the same machine failed — all rows "not measured", "engine not read"; that folder is deliberately not part of this PR.)

**Not run:**

- Correctness checks (needle recall, coding tasks): suite 1 is speed-only, so answer quality stays unverified.
- Concurrency (`--clients`) and the 32K prompt (`--long`): outside suite 1.
- Background-load recording: not captured; the GPU power/draw snapshot was taken after the run only.

## Limits

- Speed only; no correctness check was run, so the numbers say nothing about answer quality.
- The chat decode range is wide (139-332 tok/s over 9 short runs) with `--spec 4 --mtp rt` on; per-run numbers are in the README.
- Commit hash not recorded: the run happened in a working folder whose git metadata was broken, so the report says "commit not a git checkout"; the version line names project tools 0.1.61 and engine 0.1.59.1.
- Privacy: the benchmark tool replaced home folders, user/host names, local addresses and API keys before writing the folder; I read the report once before submitting — only hardware facts, launch parameters and `127.0.0.1` remain.

Related on this site

Quick links to related pages.