#5599 [Docs] reproducible output: add the CPU prompt share and the route tail skip, and correct both rows (#4673)
open · @slyfox1186 · 0 comentarios · En GitHub
Multi-GPUAMD / HIPNVIDIA / CUDADocumentationWindowsLinux
Descripción
## Problem and result
`docs/DETAILS.md` ("Reproducible output") tells you to add `STRATA_IQ_MT_MIN=1` and
`--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0` for byte-identical repeats. Two settings that are on by default and
can change an answer are missing: `STRATA_PREFILL_CPU_SHARE` and `STRATA_ROUTE_TAIL_SKIP`. The engine's own start lines
say so: "answers can differ slightly ... STRATA_PREFILL_CPU_SHARE=0 turns it off" (`src/prefill/prefill.cpp:2973-2974`)
and "answers differ slightly from 0.1.41 ... STRATA_ROUTE_TAIL_SKIP=0 turns it off" (`src/program/generate.cpp:12726-12727`).
`docs/OLDER_GPUS.md:278` already lists both for A/B runs.
In #4673, @Cryonic7 measured this on engine 0.1.56 (RTX 4090, Windows 11, the documented switches plus
`STRATA_ROUTE_TAIL_SKIP=0`, 7,800 expert slots in each start): with the CPU share at its default, the ~1K-token
repeats gave 4 and 5 distinct texts out of 5 in two starts; with `STRATA_PREFILL_CPU_SHARE=0` added, all 7 answers were
identical across the two starts. They also found that two starts with 7,800 and 7,641 slots matched on 5 of 7 answers,
and on 7 of 7 once both had the same count. Those are their measurements; I did not repeat them.
The two table rows for these settings were also out of date:
- `STRATA_PREFILL_CPU_SHARE` said "cap 1024" and "prompt chunks under 1,024 tokens". The default limit is 3,072
(`include/strata/prefill/share_rules.hpp`, `kShareLimitNew`; `src/prefill/prefill.cpp`, `cpu_share_max`). It also said
HIP keeps it off, but a single gfx12 card has it on by default (`arm_cpu_share`, `gfx12_prefill_share_default`), with a
1,536-token limit while the fused IQ2_S / IQ4_XS prompt kernels are on (`share_rules::gfx12_share_limit`,
`fused::gfx12_iq2s_default`, on by default on Linux).
- `STRATA_ROUTE_TAIL_SKIP` listed everything under "Not applied". Most of those conditions only turn the default off:
NVFP4 experts, other HIP cards and the cache threshold all leave `STRATA_ROUTE_TAIL_SKIP=7` working, and the engine's
start lines say "STRATA_ROUTE_TAIL_SKIP=7 turns it on" for NVFP4 and the threshold (`generate.cpp:12718`, `:12722`). Only a
peer tier or helper GPU stops the skip itself (`expert_source.cpp`, `expert_pool_dispatch_multi`, `:4969`:
`d.peer == nullptr && d.remote_count == 0`), and the SYCL and Metal sources have no tail-skip code. NVFP4 was missing
from the row (it is in `docs/NVFP4.md:68`).
## Related discussion
#4673. In the thread, @Cryonic7 suggested adding both switches to this paragraph and correcting the 1,024 row. Docs only;
no default or behavior changes.
## Approach
- The recipe gains the two `env` switches and the same `--expert-cache` size in every run, and links to the
"On by default" table for where each applies. It stays worded as settings for a controlled comparison.
- The CPU share row: the 3,072 limit, the gfx12 case and its 1,536 limit, the "every expert cached" case, how
`--batch` slots work (only `STRATA_PREFILL_CPU_SHARE_BATCH=1` enables it there; `STRATA_PREFILL_CPU_SHARE` alone does
not, since `share_pool` is false, `generate.cpp:10637`), and `STRATA_PREFILL_CPU_SHARE_MAX=1024` for the old limit
(lower values count as 1,024).
- The tail skip row: "off by default" kept apart from "never applies", plus the NVFP4 case.
## Validation
On `146df3c4`.
**Passed:**
- The new `#on-by-default` link resolves to the `### On by default` heading (one heading with that name in the file).
- `git diff --check`: no whitespace errors.
- The other places that give a reproducibility recipe were checked: `docs/OLDER_GPUS.md:278` (already lists both),
`docs/RESEARCH_RUNS.md:21`, `docs/SPLIT_SERVE.md:125` and `docs/BATCHING.md:733` (records of specific runs, each naming
the settings it ran with). Left unchanged.
**Failed:**
none observed
**Not run:**
- No engine run of my own. Every condition in the two rows comes from reading the source at `146df3c4`: this is
**unconfirmed static analysis** for HIP (gfx12 and others), SYCL, Metal, NVFP4 and the peer and helper paths. The
measured evidence is @Cryonic7's in #4673 (Windows, engine 0.1.56), not mine.
## Limits
This corrects the recipe and the two rows. It does not explain every case in #4673: the original report used a layer
split (`--layer-split 22`), where the CPU share is off by default.
En el sitio
Enlaces a install, modelos, releases.