Pull requests / #1992

#1992 calibrate: measure the shared-expert stream fork on AMD (STRATA_SH_STREAM) and keep it per PC (#1272)

open · @Vop3p · 0 comentarios · En GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

## Summary

`tools/calibrate.py` gets a sixth step on AMD: it measures the shared-expert stream fork (`STRATA_SH_STREAM`) off against on, a restart each, and writes `STRATA_SH_STREAM=1` into the config's `"env"` only when it wins by more than `MIN_GAIN`. This is the per-card choice left open in #1272.

The HIP default stays off (#816, #826): an RX 6800 on Windows and 4x R9700 decode 12-45% faster without the fork. The same card family on another PC goes the other way (#1272: 2x RX 6900 XT on Linux), and the output is identical either way, so a measurement on the user's PC is the right place to decide it.

## What changed

- `tools/calibrate.py`: step 6, HIP only (`cfg["backend"] == "hip"`), after the adaptive-tier step and with everything chosen before. Both values are set explicitly (`0` and `1`), so a value the config already carried does not decide the baseline; two rates each; the existing `pick()` / `MIN_GAIN` rule and the #1337 failed-start handling. `measure()` returns the choice as `"env"`; `start_engine` takes an optional env; `apply_env()` sets the chosen variables and removes the other `ENV_KEYS`, so an older calibration's choice does not linger.
- `setup.py`: `--calibrate` writes `"env"` into the config and the settings file; a reinstall applies a saved calibration's `"env"` the same way (one saved before this step leaves `"env"` alone). Where setup offers calibration is unchanged (NVIDIA only; on AMD it is still `./setup.sh --calibrate` by hand, #566).
- `docs/AMD_HIP.md`: the `STRATA_SH_STREAM` note says calibrate measures it.
- `tools/test_calibrate.py`: four tests (the step runs on HIP only and keeps the fork only on a win; a failed start keeps the default; `apply_env`; setup writes and later removes the variable) and the `run()` test now checks that `hip` is passed. 21 of 21 pass.

## Measured

Real engine (the 0.1.42 tag, built with setup's HIP flags), 2x RX 6900 XT (gfx1030, PCIe 4.0 x8 each), Ryzen 5 5600X, Ubuntu 26.04, ROCm 10.0, GSQ-RCO IQ3_S on a layer split, a setup-style config (`--spec 4 --spec-min-p 0.5`, empty `"env"`). The whole calibration ran in 982 s with no failed start and no stall:

| step | chosen | measured |
|---|---|---|
| PCIe share | 0.00 | 62.0 against 59.2 tok/s at the probed 0.39 |
| draft floor | 0.70 | 62.3 against 61.3 at 0.5 |
| pool workers | 5 (the engine's) | 3: 60.3, 2: 56.5 |
| expert tier | every 1, 80 swaps, decay 0.97 | 63.8 against 61.6 |
| **shared-expert fork (new)** | **`STRATA_SH_STREAM=1`** | **off 62.2 / 61.3, on 67.2 / 66.6 tok/s (+8.3%)** |

Identity, separately, with a fixed protocol (`--spec 4 --spec-min-p 0.5`, 4K and 32K prompts, two each, fresh starts; the baseline reproduces its text and draft counts across starts): fork on against off gives the same text on 4 of 4 requests on the split and on one card, decode +8.7 / +9.9% (split) and +5.8 / +6.0% (one card).

Not tested: Windows HIP, other AMD cards (the step's purpose is that they decide for themselves), a CUDA build (the step does not run there).

Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030, PCIe 4.0 x8 each) / Ryzen 5 5600X, ROCm 10.0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.