#3620 [Benchmark] RTX 5080 16 GB + Core Ultra 9 285K, Qwen3.8-Flash-Next IQ3_XXS 64K: 150 tok/s decode, 4.4K tok/s prompt (Linux, v0.2.1)

Description

### What did you measure

Community benchmark from a single RTX 5080 16 GB on Linux (Fedora 44), Qwen3.8-Flash-Next **IQ3_XXS**, Strata **v0.2.1** (commit `90767252`; the bench report reads the version as 0.1.46), engine compiled locally (`./setup.sh --build`). Short version: **150 tok/s decode, ~4,400 tok/s prompt at 16K-32K**. Thank you for this project, it runs very well here.

| Test | Median (range) |
|---|---|
| chat decode | **150.0 tok/s** (126.7-170.2, n=9) |
| 4K prompt, fresh | **3,755 tok/s** (3,576-3,800), TTFT 1.08 s |
| 16K prompt, fresh | **4,396 tok/s** (4,380-4,401), TTFT 3.67 s |
| 32K prompt, fresh | **4,412 tok/s** (4,411-4,419), TTFT 7.29 s |
| agent session | cache hit 69.5 %, turn TTFT 0.97 s (0.76-1.13), decode 150.3 tok/s (124.3-240.4) |

### Reproduction status

Measured locally

### Hardware

- **GPU:** RTX 5080 16 GB (sm_120), driver 615.78.08, PCIe 5.0 x16. Tuned with LACT: memory clock +6000 (offset), core capped at 2850 MHz @ 900 mV, 360 W power limit. Not stock, so stock cards may be a bit slower.
- **CPU:** Intel Core Ultra 9 285K (8P + 16E, AVX2/AVX-VNNI, no AVX-512). Setup picked 15 CPU expert workers.
- **RAM:** 64 GB (2 x 32 GB Kingston Fury KF560C36-32) DDR5 at 6400 MT/s, 61.7 GiB visible.
- **Storage:** GGUF shards (including the n-gram table) on a Kingston Fury Renegade G5 2 TB (SFYR2S2T0, PCIe 5.0 x4 NVMe), btrfs with NOCOW on the model folder.
- **OS:** Fedora 44, kernel 7.2.11, CUDA 13.4.92, GCC 16.2.1 default with g++-15 used for the build (see below).
- **Background:** normal KDE desktop session. No other GPU workloads (the GPU was idle at 31 MiB before the run).

### How I installed and ran it

1. Downloaded the GGUFs at the pinned revision (IQ3_XXS shards 1+2 and the mmproj) with aria2c, and the MTP layer with `tools/mtp_fetch.py fetch` + `verify`. Both shards' SHA-256 match the values pinned in `setup.py`.
2. Built and installed, without starting the server:
   ```sh
   export CC=gcc-15 CXX=g++-15   # see the build note below
   ./setup.sh --build --model IQ3_XXS --gguf-dir <folder with both shards> --vision no --no-start --no-browser --yes
   ```
   Setup's choices: 64K context (its recommendation for 16 GB), int8 KV, MTP on, images off. KV streaming was turned **off** because RAM was tight (~60 GB used of 62).
3. Benchmarked with `./bench.sh --long`, which started the server from the install's config:
   ```text
   strata --pack iq3_xxs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp rt --max-context 65536 --kv int8 --pool-workers 15
   ```
   On load, the engine filled the expert cache with 5,310 experts (8.61 GiB of VRAM).

### Two notes from the install

- **Build on Fedora 44 (GCC 16 default, CUDA 13.4):** setup correctly picks `g++-15` as the CUDA host compiler, but the plain C++ files are compiled with the default GCC 16. CMake then adds GCC 15's implicit link directory (`-L/usr/lib/gcc/x86_64-redhat-linux/15`) to the final link, which resolves GCC 15's `libstdc++`, and the link fails on the GCC 16 objects:
  ```text
  libstrata_prefill.a(prefill.cpp.o): undefined reference to `std::__detail::__notify_impl(void const*, bool, std::__detail::__wait_args_base const&)'
  ... undefined reference to `std::__detail::__wait_impl(void const*, std::__detail::__wait_args_base&)'
  collect2: error: ld returned 1 exit status
  ```
  Building everything with the same compiler (`CC=gcc-15 CXX=g++-15`, fresh `build/`) fixes it. Suggestion: when setup falls back to an older host compiler for CUDA, it could set `CMAKE_CXX_COMPILER`/`CMAKE_C_COMPILER` to the same one. This is related to #2576 but not the same failure.
- FYI for anyone else on the v0.2.1 tag: `./bench.sh` still hits the `NameError: name 'port' is not defined` from #2864/#2908 there. It's already fixed in v0.2.2 and main; I patched it locally the same way.

<details><summary>Full report generated by <code>tools/strata_bench.py --long</code></summary>

# Community benchmark on NVIDIA GeForce RTX 5080

Measured on 2026-10-21 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

## Hardware and software

- GPU and VRAM: NVIDIA GeForce RTX 5080, 16 GiB VRAM
- CPU: Intel(R) Core(TM) Ultra 9 285K (24 cores, 24 threads); installed RAM: 61.7 GiB; storage: not measured
- OS: Linux 7.2.11-200.fc44.x86_64; GPU driver: 615.78.08; CUDA: not read
- Strata: project 0.1.46, commit 90767252; engine 0.1.46
- Background workloads and power limits: not recorded

## Model and configuration

- Model and quantization: qwen3.8-flash-next-iq3_xxs
- Context: 65536 tokens; parallel slots: 1
- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

```text
strata --pack iq3_xxs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp rt --max-context 65536 --kv int8 --pool-workers 15
```

## Method

Run by `python tools/strata_bench.py --long` against the HTTP API of a Strata server (started by the benchmark from the install configuration); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs
- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`)
- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

## Results

| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |
| --- | ---: | ---: | ---: | ---: | --- | --- | --- |
| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 150.0 (126.7-170.2) | 0.23 (0.20-0.25) |
| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 3755.0 (3575.9-3800.1) | not measured | 1.08 (1.07-1.14) |
| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 4396.2 (4380.3-4400.5) | not measured | 3.67 (3.67-3.68) |
| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 4411.6 (4411.3-4419.0) | not measured | 7.29 (7.27-7.29) |
| agent session, follow-up turns | 14691 (14662-14725) at the last turn | see below | up to 64 | 3 | see below | 150.3 (124.3-240.4) | 0.97 (0.76-1.13) |

Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token.

### Agent session (conversation cache)

Cache hit over the follow-up turns: 69.5 % (69.4-69.5); earlier-turn tokens read again per session: 4 (3-4) (earlier turns that the cache did not hold; the 13826 (13825-13827) new tokens of the files are read once); first turn (nothing cached) first token: 0.56 s (0.56-0.59).

| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 0 | 609 | 0 | 609 | 1057.7 | 0.588 | 113.0 |
| 1 | 4086 | 624 | 3462 | 3110.2 | 1.128 | 150.1 |
| 2 | 5619 | 4149 | 1470 | 1838.9 | 0.812 | 150.3 |
| 3 | 9052 | 5683 | 3369 | 3022.1 | 1.129 | 149.7 |
| 4 | 11809 | 9101 | 2708 | 2795.2 | 0.983 | 240.4 |
| 5 | 14662 | 11843 | 2819 | 2933.1 | 0.975 | 128.3 |

Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server).

## Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 69 s.

</details>