#1776 Community benchmark: Tesla V100-32GB, i5-10505 (6-core Comet Lake), 32 GB RAM

Description

I could not believe my eyes with those numbers!! you have done an insane job on this engine!! i really cant think straight so i asked my agent Rush to compile my findings... Feast your eyes!!

# Compiled by Rush:

Measured on 2026-10-10 by **Waleed**. A daily-use agent session on a 6-core Comet Lake
office box with a Volta card, 7 GB of background services resident, and the model
experts mapped in low-RAM mode. Decode reaches 90–100 tok/s steady; the expert cache
climbs 94.6% → 99.2% within a single conversation. Main limitation: one machine, one
session, single runs, no fixed-cap harness — every number comes from the engine's own
log lines.

## Hardware and software

- **GPU:** Tesla V100 (sm_70), 32 GB, in the OptiPlex 3090 MT's full-height **Gen 3 x16**
  slot. Power Around 200-350W. Card model V100-SXM2-32GB in an SXM2 to PCIe adapter card.
- **CPU:** Intel Core i5-10505 (Comet Lake, 6C/12T, 3.2 GHz base / 4.6 GHz turbo,
  12 MB L3, **AVX2 only — no AVX-512**).
- **RAM:** 32 GB DDR4, dual-channel, **DDR4-2666 (41.6 GB/s peak)**.
- **Storage:** Crucial P3 500 GB M.2 2280 NVMe (`CT500P3SSD8`) — QLC, DRAM-less,
  Gen 3 x4, ~500 MB/s sequential read. Model files, PLE table and packs on this drive.
- **OS:** Tiny10 (community-debloat Windows 10 image), a long-running install, not
  freshly imaged.
- **CUDA Toolkit:** 12.9.4 (CUDA 12.x, the required line for sm_70; CUDA 13 dropped Volta).
- **Strata:** `main`, cloned 2026-10-10. HEAD sha not recorded; the tip of `main` at
  clone time was `fb58e0dbc` (2026-10-08, v0.1.41 line) — confirm with `git rev-parse HEAD`.
  Release binary vs source build: not recorded.
- **Background workloads during the runs:** MSI Afterburner (polls the GPU), AnyDesk
  remote session, Chrome with 1–3 tabs, a full Open WebUI install with a PostgreSQL
  server — ~7,300 MB of RAM in total, so the engine had ~24.7 GB of RAM.

## Model and configuration

- **Model:** Swift 1.5 (UkisAI fine-tune of Qwen3.8-Flash-Next), **IQ2_XS**, chosen from
  setup's menu. **Vision off** (no `mmproj`).
- **Config:** `strata-swift-iq2_xs.json`, engine args:

```text
--expert-cache auto --prefill auto --spec 4 --max-context 131072
--kv q4_0 --resident-experts --spec-min-p 0.5
```

- **Context:** 131,072 — confirmed independently: the engine's `max` field equals
  `131072 − prompt tokens` on every turn.
- **`--resident-experts`** (low-RAM mode): IQ2_XS's experts are ~35.5 GB, above the
  ~24.7 GB free after background services, so the experts the 32 GB card does not hold
  are page-locked/mapped from the model files.
- **`--calibrate` was run** and reported changing `--pool-workers`; the final value was
  not recorded. `--pcie-frac` and the resulting `--pool-workers`: not recorded.
- Client: Open WebUI, local server `127.0.0.1:11454`, `gpu 0`.
- MTP draft acceptance, VRAM cache size, memory peak, TTFT: not measured.

## Method

One continuous agent session, 9 turns, multi-turn tool use, thinking on. All
throughput figures are the engine's own timings as printed in the log. Decode tok/s is
read from the per-stage lines (`thinking` / `answering` / `writing a tool call`), never
by dividing generated tokens by total request time. Fresh prompt tokens per turn are
`Y − previous prompt tokens − previous generated tokens`, where `Y` is the total in the
`reading the prompt: X of Y` line — that line reports **cumulative** prompt length, so
`Y` alone overstates fresh work. No state reset between turns; the conversation prefix
and the expert cache carry over.

## Results

**Decode, per turn (tok/s, engine's stage lines)**

| Turn | First stage | Last stage | Expert cache hit |
| --- | ---: | ---: | ---: |
| 1 | 51.7 | 55.0 | 94.6% |
| 2 | 68.7 | 66.9 | 97.6% |
| 3 | 66.4 | 80.4 | 98.1% |
| 4 | 79.4 | 93.9 | 98.6% |
| 5 | 86.8 | 96.0 | 98.6% |
| 6 | 80.6 | 88.4 | 99.0% |
| 7 | 98.1 | 98.5 | 99.2% |
| 8 | 83.5 | 99.0 | 98.5% |
| 9 | 77.7 | 91.8 | 99.2% |

Steady-state decode once the cache warms: **90–100 tok/s**; peak stage reading 99.0.

**Prompt ingestion, fresh tokens only**

| Turn | Cumulative `Y` | Fresh tokens | Time | Fresh tok/s |
| --- | ---: | ---: | ---: | ---: |
| 1 | 7,771 | 7,771 | 6 s | 1,295 |
| 2 | 23,355 | 15,477 | 19 s | 814 |
| 3 | 23,612 | 32 | — (no prefill line; prefix cached) | — |
| 4 | 41,878 | 17,629 | 16 s | 1,102 |
| 5 | 43,236 | 1,124 | 2 s | 562 |
| 6 | 45,722 | 2,250 | 3 s | 750 |
| 7 | 50,119 | 3,967 | 5 s | 793 |
| 8 | 56,524 | 6,105 | 6 s | 1,018 |
| 9 | 63,107 | 6,398 | 7 s | 914 |

Aggregate `done` rate (includes prompt reading, so it is not a decode number): 56.9,
67.5, 83.6, 95.6, 98.1, 88.3, 100.7, 95.5, 95.5 tok/s.

## Findings

- **Decode is host-bound, as reported on other Volta boxes, and a 6-core Comet Lake is
  enough.** 90–100 tok/s on a V100 is above every published V100 figure (54–75 tok/s on
  Xeon Gold 6130 and dual E5-2696 v3 rigs), on a box with **AVX2 only** and DDR4-2666.
- **Prefill throughput scales strongly with chunk size.** Tool results of 1–4k tokens
  ingest at 562–793 tok/s; chunks of 6k+ reach 1,018–1,295. In an agent loop, batching
  tool calls so results arrive as one large chunk is worth real wall-clock.
- **`--kv q4_0` appears to cost prefill on sm_70.** `docs/NVIDIA_V100.md` states that QSA
  prompt attention uses `prompt_attn_v70_kernel` for int8 / FP16 / K8V4 KV, and **the
  decode kernel (one query at a time) for Q4_0 KV**. Published V100 prefill numbers
  (1,123–1,251 tok/s) were measured with int8 KV. A `--kv int8` A/B on this box is the
  obvious test.
- **`--spec-min-p 0.5` is below what both published V100 sweeps kept.** The E5-2696v3
  box measured 0.50: 73.5 vs **0.70: 79.2**; the other V100 report also kept 0.70. This
  box sits at setup's 0.5 default.
- **Expert cache hit rate is the dominant decode lever on a 6-core CPU.** Turn 1 printed
  `+0.1% of the routed experts over PCIe`, i.e. 5.4% of routed experts were computed on
  the CPU pool; by turn 7 the share is ~0.8%. Hit rate rose 94.6% → 99.2% inside one
  conversation, and decode rose with it. With no AVX-512, every CPU-computed expert
  costs more per token.
- **Prefix caching carries agent loops.** Turn 3 had 32 fresh tokens and no prefill line
  at all.
- **`--resident-experts` on 32 GB RAM with 7.3 GB of services running** is the fit
  answer for IQ2_XS on a 32 GB card; the runs completed with no paging and no OOM.
- **Swift 1.5 with vision off**: the `max` values (123,293 / 107,709 / 89,186 …) are
  reasoning budgets, not thinking lengths; thinking stages were 1–397 tokens, so answers
  dominate the decode measured here.

## Correctness and limitations

- Not tested: needle recall, HumanEval, quant comparisons, MTP acceptance, memory peak,
  TTFT, vision, long-context behaviour, the `--calibrate` sweep output, PCIe utilization
  (`nvidia-smi dmon -s t`), model-file read counters.
- Single run per turn; no repeats, no median/range. Numbers are engine timings, not
  client-measured.
- One machine, one model size, one session, no fixed-cap `benchmark.py` harness.
- MSI Afterburner polls the GPU during runs; clock jitter is a plausible contributor to
  the 25–30% stage-to-stage decode swings seen on other Volta rigs.