#3563 [Benchmark] RTX 3070 16GB (modded) + DDR4 — ~60 tok/s decode with `--kv-resident` freeing VRAM for experts

Description

Real-world numbers from multi-turn agent workloads (Hermes Agent / opencode) on an older desktop platform (PCIe 3.0, DDR4, 6-core CPU). Posting mainly because the **same machine went from ~45 to ~60 tok/s median decode** after a config change, and the biggest factor looks like `--kv-resident` freeing VRAM for the expert cache. This may help other 12–16 GB card owners.

> **Note:** Observational report, not a controlled benchmark. Prompts, outputs, paths and endpoints are omitted. All numbers are parsed from the engine log line `prompt N tokens = R reused + F read in … ms (… tok/s), G generated in … ms (… tok/s), drafts accepted …`, so prefill and decode are timed separately by the engine.

Related: #3558 (RTX 4060 Ti 16GB, ~39 tok/s median decode).

## TL;DR

- **GPU:** RTX 3070 with 16 GB (memory-modded; Ampere sm_86), PCIe 3.0 — engine probe 12.9 GB/s host→device
- **CPU / RAM:** i7-8700 (6C/12T, AVX2 only) / 64 GB DDR4-3600 (2×32 GB)
- **Model:** Qwen3.8-Flash-Next IQ3_XXS (GSQ-RCO pack)
- **Engine:** Strata 0.1.46 (desktop app 0.2.1, control 0.1.46.3), native Windows 11
- **Context limit:** 262,144 tokens, KV INT8, only 32K cells per layer kept in VRAM (`--kv-resident 32768`)
- **Median decode:** **60.0 tok/s** across 106 completed requests (≥64 output tokens: 60.8 tok/s, n=86)
- **Mostly fresh prefill:** ~932 tok/s median (n=9)
- **Expert cache:** 5,158 slots / 8.37 GiB of VRAM, median decode hit rate **86%**, only ~4% of routed experts over PCIe
- **Prefix reuse:** 86% of all prompt tokens were reused

## 1. Hardware and runtime

| Component | Configuration |
|---|---|
| GPU | RTX 3070, 16 GB (aftermarket memory upgrade, stock BIOS), compute only, display on iGPU |
| GPU clocks | Memory clock locked with `nvidia-smi -lmc 7000,7000` (core not locked) — done for driver stability (TDR), not tuning |
| CPU | Intel Core i7-8700 (6C/12T, no AVX-512) |
| RAM | 64 GB DDR4-3600 (XMP), 2 × 32 GB |
| Board / bus | Z390, PCIe 3.0 x16 |
| OS | Windows 11 (native), WSL2 capped at 8 GB for the agent harness |
| Engine | Strata 0.1.46 (desktop app 0.2.0-beta) |
| Model | Qwen3.8-Flash-Next IQ3_XXS |

<details>
<summary>Runtime configuration (click to expand)</summary>

| Setting | Value |
|---|---|
| `--max-context` | 262,144 |
| `--kv` | int8 |
| `--kv-resident` | **32768** (rest of the K/V in 3.09 GiB of pinned RAM) |
| `--resident-experts` | on (35.2 GiB of experts page-locked in RAM) |
| `--expert-cache` | auto → **5,158 slots / 8.37 GiB** |
| `--vram-reserve-mib` | 768 |
| `--spec` / `--spec-min-p` | 4 / 0.70 |
| `--mtp` | on (draft head) |
| `--pcie-frac` | 0.36 (matches the startup probe) |
| `--adapt-every` / `--adapt-swaps` / `--adapt-decay` / `--adapt-async` | 2 / 80 / 0.97 / 1 |
| `--prefill` | auto (chunk 8,192) |
| Conversation cache | 8,192 MiB / 6 slots / min 2,048 tokens |
| Control vector | experimental-speed-projection, scale 1.0, layers 4–44, `project`, per-layer |
| Vision | on, **on the CPU** (6 threads) |
| Env | `STRATA_ROUTE_RESIDENT=0.5` |
| CPU pool | 5 workers + host thread (default) |
| Concurrency | single slot |

</details>

## 2. Decode

Current config, 106 completed requests over two engine sessions (Oct 20–21):

| Subset | Requests | Median tok/s | Range |
|---|---:|---:|---:|
| All (≥2 output tokens) | 106 | **60.0** | 26.1–82.7 |
| Output ≥64 tokens | 86 | **60.8** | 41.2–82.7 |
| Output ≥1,000 tokens | 13 | **63.7** | 56.6–75.9 |

### By prompt length (output ≥64 tokens)

The current config has few requests above 64K, so the second column adds 3 earlier sessions where only `--vram-reserve-mib` differed (900 / 1536 instead of 768).

| Prompt tokens | Current config: n / median | Incl. reserve variants: n / median (range) |
|---|---|---|
| 8K–32K | 57 / **60.0** | 86 / 60.8 (41.2–82.7) |
| 32K–64K | 26 / **63.9** | 57 / 64.5 (52.2–79.2) |
| 64K–128K | 3 / 60.0 | 27 / **61.8** (52.8–80.1) |
| ≥128K | 0 | 0 (longest prompt seen: 88,339) |

Unlike #3558, I see **no clear drop with prompt length** up to ~88K. Decode varies more with draft acceptance, which depends heavily on content: tool-call JSON drafts very well.

## 3. Prefill

| Subset | Requests | Median tok/s | Range |
|---|---:|---:|---:|
| ≥2K tokens, zero reuse | 9 | **932** | 723–1,060 |

Prefill is slower than #3558 (~1,205 tok/s), which fits the older CPU, DDR4 and PCIe 3.0. Requests reusing ≥90% of the prompt (n=62) had a median prefill time of 2.7 s.

## 4. Caches and speculation

| Metric | Value |
|---|---:|
| Prompt tokens reused | 86.0% (2.21M of 2.57M) |
| Requests with some reuse | 80 / 108 |
| Conversations parked / restored / borrowed | 84 / 8 / 9 |
| Evictions | 0 (2 over the wider window) |
| Skipped (below 2,048-token minimum) | 21 |
| Draft acceptance (aggregate) | 76.0% (median per request 78.5%) |
| Decode expert cache hit rate (median) | 86.1% |
| Routed experts over PCIe (median) | 4.0% |
| KV streaming: block reads served from VRAM | 99.8% |

With `--kv-resident 32768`, almost all K/V reads still hit VRAM at these prompt lengths, so moving most of the KV to RAM cost very little.

## 5. Before and after on the same machine

Same hardware and model. The earlier sessions used a simpler config:

| | Before | After |
|---|---|---|
| Engine | 0.1.45.3 → 0.1.46 | 0.1.46 |
| `--kv-resident` | — (KV in VRAM) | 32768 |
| `--vram-reserve-mib` | 2048 | 768 |
| Free VRAM seen by expert cache | 6.34 GiB | **9.30 GiB** |
| Expert cache | 2,158–2,538 slots (3.5–4.2 GiB) | **5,158 slots (8.37 GiB)** |
| `--spec-min-p` | 0.5 | 0.70 |
| Adaptive tier | sync | async, 80 swaps / 2 windows |
| Conversation cache | off | 8 GiB / 6 slots |
| Speed-projection control vector | off | on |
| **Median decode (≥64 out)** | **46.2 tok/s** (n=30) | **60.8 tok/s** (n=86) |
| Decode expert hit rate (median) | 72.5% | 86.1% |
| Experts over PCIe (median) | 8.4% | 4.0% |
| Draft acceptance | 66.0% | 76.0% |
| Fresh prefill (median) | 814 tok/s (n=17) | 932 tok/s (n=9) |

**Several things changed at once and the workloads differ, so this is not an A/B test.** The log does show the mechanism for the biggest one, though: keeping only 32K KV cells per layer in VRAM plus a smaller reserve roughly **doubled the expert cache**, and the hit rate rose from ~72% to ~86%. On a 16 GB card running a model whose experts mostly live in RAM, that seems to matter more than anything else.

### Suggestions for other 12–16 GB users (to try, not proven)

1. Try `--kv-resident 32768` (or similar) and check that `expert cache auto: X GiB free` increases at startup.
2. Lower `--vram-reserve-mib` carefully — see the caution below.
3. `--adapt-async 1` with `--adapt-every 2 --adapt-swaps 80`.
4. Run vision on the CPU if you rarely send images, so it does not use VRAM.
5. Compare `--spec-min-p 0.70` with 0.50 on your own workload.

## 6. Caution: VRAM headroom

With `--vram-reserve-mib 768`, the engine warns that **only 536 MiB of VRAM is free** at request start (below its 1,024 MiB guideline, #2065), meaning the driver may page GPU memory and decode can collapse. I have not seen that during these sessions, but this is close to the edge. On a card that also drives a display, use a larger reserve.

## 7. Limitations

- Real agent workloads with varying prompts, outputs and tools; no controlled A/B per setting.
- No requests above ~88K tokens yet, although the context is configured for 262K.
- Only 9 zero-reuse prefill samples in the current config.
- Draft acceptance (and so decode) depends heavily on content; agent tool calls may draft better than prose.
- The card is a non-standard 16 GB memory mod of the RTX 3070.
- No TTFT, quality or concurrency measurements (single slot).