#2342 RTX 3080 10 GB / 64 GB RAM: KV residency, expert cache capacity, and benchmark observations

Beschreibung

While benchmarking **Qwen3.8-Flash-Next IQ3_S** on **Strata** (RTX 3080 10 GB, **64 GB system RAM**, 131,072 max context, INT8 KV), I noticed a substantial improvement after changing the memory configuration. I then asked **GPT-6.1 Sol** to inspect the relevant source code and explain the results.

This is a **benchmark observation and code-analysis summary**, **not a confirmed bug report**.

## Benchmark observations

| Configuration | Initial GPU expert cache | Prefill at ~130K input | Decode at ~130K |
| --- | ---: | ---: | ---: |
| Original automatic allocation (no explicit `--kv-resident`) | 756 experts / 1.46 GiB | ~628 tok/s | ~49 tok/s |
| `--kv-resident 32768` (with adaptation overrides) | 1,416 experts / 2.73 GiB | 1,528.7 tok/s (median, 2 runs) | 61.5 tok/s |
| `--kv-resident 16384` (with adaptation overrides) | 1,491 experts / 2.88 GiB | 1,608 tok/s (1 run) | 61.2 tok/s |
| `--kv-resident 8192` | 1,491 experts / 2.88 GiB | Essentially unchanged from 16K | Essentially unchanged |

The 32K and 16K measurements used almost identical prompts (~129,973 vs. ~129,970 actual tokens), with 256 generated tokens. Moving from 32K to 16K improved measured prefill by **~5.2%**. Additional experiments changing `--adapt-swaps` (240 → 80) and `--pcie-frac` (0.00 → 0.20) showed no meaningful improvement in this test.

During the initial tuning, I observed approximately **2.2× higher prefill throughput** compared with the starting setup. **This is not yet a controlled 1:1 comparison:** the initial measurement was at a different prompt length, and multiple settings changed. The most reliable directly observed difference is the expert-cache allocation increase from **756 to 1,416 experts** after the first configuration changes.

## Follow-up source analysis by GPT-6.1 Sol

After seeing these results, I asked GPT-6.1 Sol to analyze Strata's setup, allocation, and calibration code. Its findings were:

- **KV allocation is the likely explanation for the cache increase (high confidence).** Setting `--kv-resident 32768` reduces main-model and MTP GPU state by approximately **1.2705 GiB**, closely matching the observed **~1.27 GiB** increase in GPU expert-cache allocation. Adaptive flags affect expert replacement behavior, not initial cache capacity.
- **Setup's RAM gate matters (high confidence for the inspected source).** The IQ3_S / 131K setup path requires approximately **64.799 GiB**, above the RAM available on this **64 GB** machine. This conservative requirement is intentional and has already been discussed in [issue #620](https://github.com/Niko1221/Strata/issues/620). The original setup version/logs would be needed to confirm its exact role in my initial configuration.
- **Calibration does not appear to be the cause (high confidence).** The inspected calibration code tunes decode scheduling/adaptation, not KV residency or initial expert-cache capacity. The analysis did not identify a calibration defect explaining this result.

**Working hypothesis:** on some memory-constrained configurations, a lower GPU KV residency can free enough VRAM for a significantly larger expert cache, potentially improving prefill throughput. This does **not** establish that the automatic allocator is incorrect or that lower residency is universally better.

## Scope and next steps

These numbers come **only from the synthetic benchmark**, **not from a live Claude Code / agentic workflow**. They do not yet establish improvements in multi-turn prompt reuse, tool-heavy sessions, or sustained real-world use.

They also apply specifically to my **64 GB RAM** configuration. **Where possible, I will check whether similar gains appear with larger context workloads and with 128 GB of system RAM.**

The most useful next controlled comparison would be the **original configuration vs. the original configuration plus only `--kv-resident 32768`**, using identical ~16K inputs, generation limits, and repeated runs. It would help separate the effect of KV residency from the other tuning changes.

I am sharing this mainly to ask whether this memory trade-off is expected, and whether setup could make the KV-residency / expert-cache trade-off more visible or offer an alternative profile for limited-VRAM GPUs.