#3558 [Benchmark] RTX 4060 Ti 16GB — Long-Context Performance on Windows

本文

# Benchmark: RTX 4060 Ti 16GB — Long-Context Performance on Windows

Sharing real-world performance observations from multi-turn agent workloads running Strata 0.1.46 on Windows 11.

> **Note:** This is an observational report, not a standardized benchmark or controlled A/B test. Prompts, outputs, project details, endpoints, and local paths have been omitted.

## TL;DR

- **GPU:** RTX 4060 Ti 16GB
- **CPU / RAM:** i5-13400 / 64GB DDR5-6000
- **Model:** Qwen3.8-Flash-Next IQ3_XXS
- **Engine:** Strata 0.1.46 (native Windows 11)
- **Context limit:** 204,800 tokens (configured, not fully tested)
- **Median decode:** 39.1 tok/s across 158 completed requests
- **Mostly fresh prefill:** ~1,205 tok/s median
- **Longest observed prompt:** ~140K tokens
- **Conversation prefix reuse:** 68.5% in a separate cumulative snapshot

The main observation is that long-context inference remained usable on this consumer GPU, while host-memory pressure could limit conversation-cache retention.

## 1. Hardware and Runtime

| Component | Configuration |
|---|---|
| GPU | NVIDIA RTX 4060 Ti 16GB |
| CPU | Intel Core i5-13400 |
| RAM | 64 GiB (2 × 32 GiB DDR5-6000) |
| OS | Windows 11 (native) |
| Engine | Strata 0.1.46 |
| Model | Qwen3.8-Flash-Next IQ3_XXS |
| Background load | Normal desktop applications, including Chrome |

Driver versions, GPU clocks, power limits, and peak memory consumption were not recorded.

<details>
<summary><b>Runtime configuration (click to expand)</b></summary>

| Setting | Value |
|---|---|
| Maximum context | 204,800 tokens |
| KV format | INT8 |
| GPU expert cache | 3,855 slots / 6,437 MiB |
| Host expert arena | 37,939 MiB (~37.05 GiB) |
| CPU pool workers | 9 |
| `spec` | 6 |
| `mtp_max` | 4 |
| `lookup` | 3 |
| `spec_min_p` | 0.50 |
| `pcie_frac` | 0.36 |
| Prefill cap | 8,192 |
| Conversation cache | 8,192 MiB / 4 slots |
| Free-memory floor | 2,560 MiB |
| Vision / speed projection | Disabled |

Startup logs confirmed that the host expert arena was page-locked and mapped.

</details>

## 2. Measurement Method

The preserved monitor table contains **160 requests**:

- 158 completed (`Done`)
- 2 stopped (`Stopped`)

Prompts, output lengths, and sampling conditions varied. Conversation prefixes were frequently reused.

Throughput figures come from the engine monitor's separate prefill and decode timings, rather than dividing output tokens by total request duration.

No client-side time-to-first-token (TTFT) measurements were collected.

All medians are calculated across individual requests. Reported ranges are observed minimum and maximum values, not confidence intervals.

## 3. Decode Performance

### Overall throughput

| Completed requests | Count | Median tok/s | Observed range |
|---|---:|---:|---:|
| All | 158 | **39.1** | 18.2–59.1 |
| Output ≥64 tokens | 137 | **39.4** | 30.9–59.1 |
| Output ≥1,000 tokens | 14 | **38.25** | 31.9–47.9 |

Longer-output subsets help reduce the influence of very short responses, but do not replace the overall result.

### Decode throughput by prompt length

Only completed requests with at least 64 output tokens are included.

| Prompt tokens | Requests | Median tok/s | Range |
|---|---:|---:|---:|
| <32K | 26 | 45.85 | 33.3–56.7 |
| 32K–64K | 60 | 40.95 | 30.9–59.1 |
| 64K–128K | 47 | 38.3 | 31.6–43.5 |
| ≥128K | 4 | 38.95 | 35.0–41.8 |

Decode throughput generally decreased at longer prompt lengths, but remained around 38–39 tok/s in the observed higher-context groups.

**Caution:** These are different real-world requests, not controlled experiments with identical workloads at different context lengths. The ≥128K group also contains only four observations.

## 4. Prefill Performance

| Completed requests | Count | Median tok/s | Range |
|---|---:|---:|---:|
| No prompt reuse | 13 | **1,062** | 769–1,132 |
| ≥80% fresh input | 36 | **1,204.5** | 769–1,266 |

The zero-reuse group is included in the mostly-fresh group.

Zero prompt reuse does not necessarily imply a cold GPU expert cache.

Another 81 completed requests reused at least 90% of their input tokens. Their small fresh tails make reported prefill throughput sensitive to fixed overhead, so these requests were excluded from the headline fresh-prefill estimate.

## 5. Conversation Cache

A separate cumulative session snapshot reported:

| Metric | Observation |
|---|---:|
| Cumulative input tokens | ~10.2 million |
| Token reuse ratio | **68.5%** |
| Requests with some prefix reuse | 150 / 165 |
| Cache evictions | 89 |
| Aggregate draft acceptance | ~60.6% |

These counters cover a different observation window from the 160-request benchmark table and should not be combined with it.

### Memory pressure and snapshot retention

One log entry explicitly showed a conversation snapshot being skipped because the available physical memory was insufficient.

Approximate memory requirements:

- Additional snapshot memory: **2.7 GiB**
- Required free-memory floor: **2.5 GiB**
- Available memory: **3.0 GiB**

This demonstrates that physical host-memory availability can prevent snapshot retention even when a nominal conversation-cache budget is configured.

It does **not** establish that all 89 evictions were caused by memory pressure.

## 6. Follow-up: Increasing Conversation Cache to 16 GiB

The conversation-cache budget was subsequently increased from **8 GiB to 16 GiB**, while retaining:

- 4 cache slots
- 2,560 MiB free-memory floor

Restart metadata confirmed that the new budget was applied.

Three subsequent requests were recorded:

| Prompt | Reused | Output | Prefill | Decode | Duration |
|---:|---:|---:|---:|---:|---:|
| 139,762 | 137,433 | 491 | 438 tok/s | 38.9 tok/s | 18.0 s |
| 70,552 | 0 | 165 | 1,038 tok/s | 38.6 tok/s | 72.3 s |
| 139,917 | 137,433 | 8,899 | 449 tok/s | 40.6 tok/s | 224.5 s |

The observations demonstrate successful generation and substantial prefix reuse at approximately 140K prompt tokens.

However, they **do not establish a performance improvement from increasing the cache budget**, because the workloads were not repeated under matched conditions.

## 7. Limitations

- Non-standardized workloads with variable prompts, sampling, output lengths, and background applications.
- No controlled comparison across context lengths.
- No standardized quality, recall, tool-call correctness, or concurrency tests.
- No client-side TTFT measurements.
- No controlled A/B test of the 8 GiB versus 16 GiB cache budgets.
- No controlled comparison of projection modes or `STRATA_ADAPT_OVERLAP`.
- Two stopped requests were excluded from successful-throughput statistics; their status alone does not imply an engine failure.
- No project content, prompts, outputs, local paths, or operational identifiers are included.

## 8. Summary

On this Windows 11 system with an RTX 4060 Ti 16GB and 64GB host RAM, Strata 0.1.46 delivered:

- **~39 tok/s median decode throughput**
- **~1.2K tok/s median mostly-fresh prefill throughput**
- Successful generation with approximately **140K prompt tokens**
- Substantial conversation-prefix reuse in multi-turn agent workloads

The main practical constraint observed was not simply the GPU's 16GB VRAM capacity, but the interaction between GPU expert caching, host-memory allocation, and conversation-snapshot retention.

These results suggest that long-context inference with a heavily quantized model can be practical on this hardware, although performance, cache behavior, and reliability under controlled workloads remain to be evaluated.