#6074 [Engine] 0.1.60 prompt kernels: -13% at 20,000 tokens on a 16 GB card, +8% at 40,000, flat at 16,000

open · @ShaggyDragon · 39 comments · View on GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

**Area:** Engine
**Desktop app, Terminal or source?:** Source engine package, headless in Docker (no app, no Terminal)
**Reproduction status:** Reproduced. Three prompt sizes, 5 to 10 reps each, interleaved with a same-binary drift control.
**Strata version:** engine 0.1.60 from `strata-engine-0.1.60-linux-x86_64-cuda-cuda13.tar.gz` (v0.2.14 asset) against engine 0.1.59 in `ghcr.io/niko1221/strata:0.2.13-cuda13`
**Graphics card, memory and system:** RTX 5080 16 GB (sm_120), CUDA 13, 2 x Xeon 8176, 256 GB DDR4-2133, Unraid 7.3, Docker, no host compiler

Thank you for shipping the engine as a self-contained package, and for reporting the engine version in `/metrics`. That is what made this test possible at all: there is no 0.2.14 container image on GHCR yet (the registry stops at `0.2.13-cuda13`), so I bind-mounted the released 0.1.60 engine directory over `/opt/strata/components/engine/0.1.59-linux-x86_64-cuda-cuda13` inside the normal container. The launcher, `serve.server`, the socket, the preset and every argument stayed production-identical, and `/metrics` `version` confirmed 0.1.60 was live.

## What I measured

Prompt read rate, tokens per second, medians. Same prompts, same pack, same arguments, only the engine binary differs. The A/B/A ordering prices drift on the unchanged binary.

| prompt | 0.1.59 | 0.1.60 | change |
|---|---|---:|---:|
| 16,000 tokens (one chunk) | 3,010 | 3,027 | **+0.6%** |
| 20,000 tokens (two chunks) | 2,619 / 2,623 | 2,281 | **-13.0%** |
| 40,000 tokens (three chunks) | 2,659 | 2,881 | **+8.3%** |

Drift on the unchanged binary, same session: +0.2% and -1.8% across two brackets. The 20,000-token result is 10 clean reps on each side, ranges 2,474-2,661 (0.1.59) against 2,226-2,351 (0.1.60): they do not overlap. Decode is flat in every block.

## Why it looks size-dependent, not kernel-wide

The 40,000-token gain is in the same direction as your published rows. The 20,000-token loss is not, and the only structural difference at that size is the trailing chunk: 17,664 + 2,336 tokens on 0.1.59, 17,920 + 2,080 on 0.1.60. At 40,000 the trailing chunk is 4,672 tokens, and at 16,000 there is no partial chunk at all.

Hypothesis, not yet tested: the new prompt path has a slower route when it reaches a small final chunk, and 20,000 tokens lands in that hole. If that is the case the loss would not show up in a benchmark built on whole-chunk sizes.

Two other differences from the same runs, for information: `prefill_cap` under a bare `--prefill auto` is 17,664 on 0.1.59 and 17,920 on 0.1.60, and `expert_slots` is 3,886 against 3,899.

## Configuration

`--pack iq3_s --native ...GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf ...-00002-of-00002.gguf --expert-profile <bin> --kv int8 --vision --adapt-async 1 --conversation-cache-mib 65536 --conversation-cache-slots 8 --expert-cache auto --kv-resident 20480 --max-context 262144 --mtp <rt> --pcie-frac 0.00 --ple-io ram --pool-workers 27 --prefill auto --prompt-cache-every 4096 --short-read 0 --spec 4 --spec-min-p 0.7 --vram-reserve-mib 1408`

Temperature 0, `max_tokens` 300, unique 20,000-token and 40,000-token prompts per rep, no conversation reuse. Every rep was gated on `/status` reporting `busy=false` and `queued=0` immediately before and after it, and a rep that saw foreign traffic was retried rather than averaged in — one chat request during an early pass halved a rep from 2,300 to 1,086 t/s, which is how I know the gate matters on `parallel=1`.

I have not adopted 0.2.14 here: a 13% loss on one size with no gain at another is not a trade I can make on the production brain. I am happy to run more sizes (18,000 and 24,000 would separate the trailing-chunk hypothesis from everything else), repeat on a second card, or test a 0.1.61 build the same way.

Related on this site

Quick links to related pages.