Pull requests / #2176
#2176 STRATA_ROUTE_PRIOR_MAP: the route prior per layer (opt-in); off in layers 0-11 keeps the speed with KL 0.141 -> 0.108
open · @zsariboga · 0 comentários · No GitHub
BenchmarksNVIDIA / CUDAModels & quantsDocumentation
Descrição
## What `STRATA_ROUTE_PRIOR_MAP=<one digit per layer>` (opt-in, with `STRATA_ROUTE_PRIOR` > 0, changes the output like the prior itself): the route prior's bonus per layer is digit / 10; `0` turns the prior off for that layer, layers past the end of the map keep `STRATA_ROUTE_PRIOR`. Unset: the code path is the same as today. ## Why `docs/DETAILS.md` lists `STRATA_ROUTE_PRIOR` as "adds speed, doubles the KL". Most of that KL comes from the early layers: a swapped expert there changes everything after it. Measured with the prior (0.5) on in one band of 12 layers at a time (KL against the same build with no prior, 47 prompts, Turkish / English / code, teacher-forced): | prior 0.5 only in layers | KL | | --- | --- | | 0-11 | 0.111 | | 12-23 | 0.072 | | 24-35 | 0.069 | | 36-47 | 0.057 | | all (no map) | 0.141 | ## Measured RTX 5070 Ti Laptop 12 GB, Core Ultra 7 255HX, Swift 1.5 IQ3_XXS (GSQ-RCO), `STRATA_ROUTE_PRIOR=0.5`, decode speed from agent turns in mirrored order (engine generation tok/s): | map | KL all / code | decode tok/s | | --- | --- | --- | | none (every layer 0.5) | 0.141 / 0.198 | 113.9 | | `000000000000` + 36 x `5` (off in layers 0-11) | **0.108 / 0.147** | 113.0 (-0.7%, run-to-run spread is ~3%) | | layers 12-23 and 36-47 only | 0.095 / 0.128 | 108.1 (-5.0%) | KL is against the same build and config with no prior (every change here has a floor of ~0.04-0.05 from the other opt-in switches of the config, so the bands do not add up). So the first map keeps the prior's speed with about a quarter less KL (a third less in code). In daily use since 12 October (`STRATA_ROUTE_PRIOR=0.5`, `STRATA_ROUTE_PRIOR_MAP=000000000000555555555555555555555555555555555555`). Built and smoke-tested on today's `main` (a0116b27). `docs/DETAILS.md` gets the table.
No site
Links install, modelos, releases.