#2482 STRATA_ROUTE_TAIL_SKIP=7 vs 0 on an RTX 5080 16 GB (0.1.44, UD-IQ4_XS, 352K real conversation): +6% warm decode, prefill unchanged; the decode hit rate excludes skipped experts

Descrição

### Setup

- RTX 5080 16 GB (sm_120), Xeon w7-2475X, 128 GB DDR5, Windows; engine **0.1.44** (not 0.1.44.1)
- Unsloth UD-IQ4_XS, `--kv int8`, `--rope-scaling yarn --rope-scale 3 --max-context 786432`, `--expert-cache auto` (2401 slots, 5.40 GiB), `--vram-reserve-mib 1420`, `--spec 5 --spec-min-p 0.8`, `--suffix-draft 0`, `--pcie-frac 0.33`, `--resident-budget-gib 55`
- env: `STRATA_IQ_MT_MIN=1 STRATA_IQ_GATHER=1 STRATA_IQ_PREFETCH=4096 STRATA_STAGER_THREADS=8 STRATA_PREFILL_CPU_SHARE=0`
- The two arms differ only in `STRATA_ROUTE_TAIL_SKIP`: `0` vs unset (the engine logs `STRATA_ROUTE_TAIL_SKIP=7 is ON (default)`).

### 1. Fixed-token benchmark: 352,130-token real conversation, temperature 0, 512 tokens

A conversation exported from Jan (the desktop chat app, jan.ai, which I use as the OpenAI-compatible client for Strata): 109 messages, Italian, code, tool calls, plus a fixed question; fresh engine, one cold read, then runs that reuse the 352,123-token prefix (decode only). All numbers below are from the engine's own `strata serve: prompt ...` lines.

| arm | prefill tok/s (cold) | decode tok/s (cold) | decode tok/s (warm runs) | drafts accepted (warm) |
|---|---|---|---|---|
| TAIL_SKIP=0, session 1 | 2271.6 | 56.7 | 57.2 | 221 / 307 |
| TAIL_SKIP=0, session 2 | 2282.8 | 52.8 | 55.8 | 221 / 307 |
| TAIL_SKIP=7 | 2283.5 | 54.0 | 58.2, 62.0, 59.9 | 223/345, 224/348, 210/333 |

- Warm decode: mean 60.0 vs 56.5 tok/s, **+6.2%** (60.03 / 56.5). All three skip runs are above the best no-skip run.
- Prefill: unchanged (2283.5 vs 2271.6 / 2282.8), consistent with the skip acting only in verify windows.
- Cold decode: 54.0 sits between the two no-skip values (52.8 / 56.7), so no readable difference.
- Limits: only two no-skip warm runs (one per session) and three skip runs.

### 2. Real chat use (Jan desktop app as client, temperature 1, long answers)

Requests with more than 4K output tokens, from the same logs:

| arm | context at start | output tokens | drafts accepted | decode tok/s |
|---|---|---|---|---|
| skip | 200K | 18,189 | 86.2% | 67.7 |
| skip | 224K | 13,526 | 83.3% | 60.7 |
| skip | 239K | 20,947 | 83.6% | 58.3 |
| skip | 260K | 6,023 | 87.5% | 60.4 |
| skip | 266K | 7,488 | 79.5% | 64.4 |
| no skip | 191K | 31,754 | 84.8% | 62.3 |
| no skip | 224K | 7,775 | 87.4% | 58.1 |
| no skip | 232K | 4,739 | 71.9% | 59.0 |

Same context (224K): 60.7 vs 58.1 (+4.5%) with lower draft acceptance on the skip side. Content differs between requests, so this is only consistent with section 1, not a measurement of its own.

### 3. The decode hit rate is not comparable between the two settings

In `decode expert cache hit rate: X% (hits / lookups); N more read ... (Y% of all R routed)`, the `routed` total is a multiple of 480 in **129 of 129** request lines with `TAIL_SKIP=0`, and in **0 of 16** lines with `TAIL_SKIP=7`. My reading (not checked in the code) is that skipped experts are dropped from both `lookups` and `routed`. The hit rate then rises because the denominator shrinks, not because the cache serves more experts. On the 352K benchmark it reads 63-66% with the skip and 56-58% without.

This matters whenever hit rates are compared across settings (as in the original #2441 report). Reporting skipped experts in the same line (or computing the hit rate over all routed experts) would make them comparable.

### 4. Output quality (anecdotal)

Two paired generations at temperature 1 (same prompt, one per arm): a ~477K-context conversation summary checked number by number against the conversation, and a 4-part test (two exact-answer math questions, a look-ahead bug in a pandas z-score, and a hardware question with no available data), judged blind. I saw no degradation in either pair. This only rules out an obvious problem, not a small one.

### Summary

On a 16 GB card at 200-350K context, the default `TAIL_SKIP=7` gave roughly +4-9% decode in use and +6% on the fixed-token run, with no visible quality loss. I am keeping it on. Happy to run more A/B on this machine if useful, for example several no-skip warm runs within one session, or other ranks.