#3560 [Engine] Four RTX 4090: v0.2.1.4 increases 128K cache-hit TTFT with 16 slots

Description

### Area

[Engine] — serving latency, four-GPU layer split.

### Desktop app, Terminal or source?

Source builds in Docker, using the Python OpenAI-compatible server.

### What happened

A same-day comparison of engine 0.1.45.3 against v0.2.1.4 source engine 0.1.46.3 reproduced a cache-hit TTFT regression on four RTX 4090 cards. Both arms used the same weights, parameters, container image and frozen requests. Each arm began after a process restart and the same warmup.

Three 128K repeat pairs reused identical prefixes and read only seven new tokens:

| Repeat | cache_n, both | prompt_n, both | old prompt_ms | new prompt_ms | old TTFT s | new TTFT s |
|---|---:|---:|---:|---:|---:|---:|
| 0 | 130649 | 7 | 876.6 | 1555.2 | 0.940 | 1.619 |
| 1 | 130647 | 7 | 872.8 | 1551.9 | 0.933 | 1.616 |
| 2 | 130657 | 7 | 866.9 | 1566.1 | 0.929 | 1.630 |

Median cache-hit TTFT rose **0.933 -> 1.619 s (+73.6%)**, while native decode improved **199.8 -> 238.3 token/s**. Input/output hashes and token counts match. The extra wait is present in native `prompt_ms`; this is not explained by generating a longer answer, but its code-level cause has not been profiled.

For 16 simultaneous 192-token short replies, p50 TTFT rose **3.526 -> 4.743 s (+34.5%)** and aggregate output/group wall throughput fell **256.07 -> 208.17 token/s (-18.7%)**. All three groups in each arm had clean request accounting.

Expected: cached requests should not acquire this additional prompt-phase delay after the upgrade with unchanged settings. These are observations on this machine, not claims for other GPUs.

### Smallest steps or command that reproduce it

Local reproduction used a frozen 64-request sequence per arm: one warmup, then three repetitions of single, 16-client, 128K first/repeat and 256K first/repeat. Long requests return six exact lookup values. Each first/repeat pair sends the same JSON payload twice; the second response reports the cache/prompt counts above. The measured client runs on the inference host, disables proxies and rejects redirects.

The associated results-only PR supplies the 52 short-request payload references, all per-request timings/usage/hashes, configuration, source/build provenance and group accounting. The 12 long-context payload references contain private network/path material and are withheld. Their original hashes, token counts and retrieval-pass evidence are included, but this prevents public byte-for-byte replay of the long inputs. A new synthetic long prompt would be a different experiment; it must not be assigned the recorded hashes.

### Reproduction status

Reproduced locally with real model requests. Both arms completed 64 requests; all replies passed their correctness/length checks. No arm was rerun to improve the result.

### Strata version

Old: engine 0.1.45.3, source integration `207fa05843fbec81e3a5fc5fe069ebda8ab0ff30`.

New: v0.2.1.4 upstream `9de04cd44e927a9a3df6bbc0e94fc1be8b844ba4`; [native build tree](https://github.com/oscarlius/Strata/commit/1df49d07b413c415c926cf98ae9270bed73daf8a), [served tree](https://github.com/oscarlius/Strata/commit/1e0083effde9941606318166d66fc224b3af8626), engine 0.1.46.3. Both retain the five service/metrics fixes; those do not alter native inference code. The served tree adds a two-line Web display guard after the native build; native/Python inputs are identical.

### Graphics card, memory and system

4 x RTX 4090, 24,564 MiB each, 200 W each, driver 595.99.02; 2 x Xeon Platinum 8380 (80 physical cores / 160 threads), 251.3 GiB RAM, Linux 6.8.0-100/glibc 2.39. Same CUDA 13 container image and sm_89 build options; llama.cpp pinned to `3cf03257f219afbe7334045ff7c6a06ac68c627d`.

### Model, size and settings

Qwen3.8-Flash-Next GSQ-RCO IQ3_S, existing native pack, Q2_0 MTP, spec 4 / min-p 0.5. GPUs 0,1,2,3, layer boundaries 12,24,36, 16 slots / four groups, prefill 2048, context 262144, int8 KV / resident 24576, pictures enabled, expert cache auto, PLE IO RAM, pool workers 0, GPU reserves 1536 MiB. Prompt cache 1 / every 0 / root 0; conversation cache 98304 MiB / 1024 entries / minimum free 32768 MiB. `STRATA_PIPED_CONCURRENT=1`; self-commit and prior experimental flags absent. Weight revisions/full-file hashes were not collected in this diagnostic; weights were not changed.

### Evidence and limits

The second new 256K first-read group had extra requests or accounting uncertainty and mismatched cache/prompt counts. Its subsequent repeat may also have a different restored-state path. Both 256K categories are therefore **INCONCLUSIVE** for the planned three-pair claim; the raw data and conservative exclusions are retained.

This is a sequential same-day custom diagnostic, not standard-v1, an interleaved A/B/A or a full quality benchmark. Three repetitions are not confidence intervals. Short single-request TTFT varied substantially and is weaker evidence. No runtime tuning or speculative fix is proposed. Related long-context discussions include #2275 and #3351, without asserting the same root cause.

Results-only evidence PR: https://github.com/Niko1221/Strata/pull/3562