#1913 Decode speed is bistable within one session: per-layer host per expert entry jumps 3-5x and stays there, while entries / GPU-reach wait / CPU kernel time stay constant

本文

Environment (short)
- Strata 0.1.42 (tag v0.1.42 commit 61b3fb5, v0.1.40, v0.1.41), CUDA build
- RTX 3080 20 GB (sm_86), single GPU, driver 616.92
- Xeon E5-12C/24T, AVX2 (no AVX-512), Win 10, 96 GB RAM
- Qwen3.8-Flash-Next IQ2_XS, native pack
- Flags: --spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 0 --max-context 262144 --vram-reserve-mib 700 --host-core last --pool-workers 12
- Env: STRATA_DECODE_TIMING=1, STRATA_POOL_SPIN_US=20000, STRATA_PREFILL_CPU_SHARE=0
- Expert arena on 2 MB large pages; route tail skip ON (default)
(STRATA_PREFILL_CPU_SHARE=0 is deliberate: no AVX-512 here, this is #1729 - per-layer host CPU term 15.7 -> 71.5 ms with the share on.)

1. What happens
11 consecutive requests inside one engine process, no config change between them:
  #1-6:   91.4 - 99.8 tok/s, per-layer host 17.9-21.8 ms, entries/window 4.76-5.43, host per entry 3.4-3.9 ms
  #7-11:  20.1 - 44.7 tok/s, per-layer host 61.1-137.2 ms, entries/window 4.99-5.30, host per entry 11.3-15.7 ms
Between request 6 and 7 the speed drops 98.2 -> 44.7 tok/s and never recovers for the rest of the session.

2. What does NOT change across the same 11 requests
- entries/window: 4.76-5.43 before, 4.99-5.30 after (same work handed to the CPU pool)
- GPU-reach wait: 12.72-13.75 ms throughout (GPU is not the variable)
- CPU kernel time: 3.82-4.24 ms at ~5 entries on both sides
- avg T (draft acceptance): 3.72-3.92 on both sides
- plan / actq / jobs in the [plan actq jobs CPU] breakdown: all <= 0.5 ms in every row
So the jump is essentially all in the CPU term - the pool dispatch wall clock - not in compute, and not in lock contention inside the engine. The pool takes 3-5x longer to hand back the same number of experts, and stays that way.

3. For scale
Over ~180 requests on this machine, bucketed by expert-cache hit rate, the normal cost of one missed expert is 3.4-5.1 ms of host time, and speed tracks the hit rate as a step. In section 1 the same machine and build show 11-16 ms per entry - the per-entry cost itself tripled. No amount of cache would have fixed that.

4. Three things I noticed in the same window (may be unrelated)
(a) The PCIe probe varies a lot between starts on an idle machine: 12.4, 9.0, 9.1 GB/s host->device, with individual samples from 4.5 to 12.4, moving pcie_frac 0.25 -> 0.34. It is the number the new adaptive pcie_frac starts from.
(b) VRAM is nearly exhausted: "389 MiB of VRAM free with everything loaded", and "strata verify: capturing the 5-token window (235 MiB of VRAM free)" appears in the lines immediately before request 7, the first slow one. Causal or coincidence, I cannot tell - the timing lines up.
(c) Widening --max-context 131072 -> 262144 costs the expert cache ~17% of its slots (7661 -> 6291) on the same model, since --kv-resident 0 keeps the whole KV resident.

5. Possible bug: --pool-workers 12 and 14 both print only 11 pinned processors
With --host-core last, every count prints its own number of logical processors except these two:
  11 pool workers on logical processors 0,2,4,6,8,10,12,14,16,18,20   (11 listed)
  12 pool workers on logical processors 0,2,4,6,8,10,12,14,16,18,20   (11 listed)  <- 12 requested
  14 pool workers on logical processors 0,2,4,6,8,10,12,14,16,18,20   (11 listed)  <- 14 requested
Workers past the 11th appear to run unpinned. Measured here that shows up as 12 ~= 14 (tied) and 16 clearly worse. Intended, or a placement bug?

6. Side note on tuning
Before the arena used large pages, --pool-workers 6 beat the default on this machine (37.1 -> 55.0 tok/s), which matches the --calibrate hint. After large pages, 12 is best (91.9 tok/s vs 80.6 at 6). If --calibrate is meant for Windows users, it may be worth having it notice whether the arena is on large pages - following the current guidance here would pick 6, which is 24-31% off the best.

7. Questions

-   1. Could the engine detect and report the section-1 jump? The counters already exist in the decode timing line: a one-line warning when "host per entry" exceeds ~2x its own session median would turn "it feels slow" into "the log says why".
-   2. Is there anything to make the pool resilient to external CPU pressure? plan/actq/jobs stay <= 0.5 ms, so it looks like the workers are simply not being scheduled. Here 11 pinned workers + 1 host occupy 11 of 12 physical cores, so any other CPU consumer lands on top of them. Would leaving a core empty (--pool-workers 10) trade peak speed for stability, or should the spin/priority already handle that?
-   3. Is the section-4(a) probe spread expected (4.5-12.4 GB/s)?
-   4. Section 5 - intended or bug?

8. Method
Everything above comes from the engine's own STRATA_DECODE_TIMING=1 line, one per request; "host per entry" = the CPU term divided by entries, both from that line. The 11 requests in section 1 are inside one process - the log has one "session is up" and no "engine started" between them. What I cannot show: the wall-clock stamps are inherited from the parent process, so I cannot align them with outside events, and I did not capture a process list while it happened. External CPU preemption is my first guess (the same signature appeared here before right after Windows Defender scanned a freshly downloaded engine), but I would rather report the numbers than the guess.

Happy to split sections 4-6 into separate issues if preferred.