Pull requests / #2176

#2176 STRATA_ROUTE_PRIOR_MAP: the route prior per layer (opt-in); off in layers 0-11 keeps the speed with KL 0.141 -> 0.108

open · @zsariboga · 0 commentaires · Sur GitHub

BenchmarksNVIDIA / CUDAModels & quantsDocumentation

Description

## What

`STRATA_ROUTE_PRIOR_MAP=<one digit per layer>` (opt-in, with `STRATA_ROUTE_PRIOR` > 0, changes the output like the prior
itself): the route prior's bonus per layer is digit / 10; `0` turns the prior off for that layer, layers past the end of
the map keep `STRATA_ROUTE_PRIOR`. Unset: the code path is the same as today.

## Why

`docs/DETAILS.md` lists `STRATA_ROUTE_PRIOR` as "adds speed, doubles the KL". Most of that KL comes from the early layers:
a swapped expert there changes everything after it. Measured with the prior (0.5) on in one band of 12 layers at a time
(KL against the same build with no prior, 47 prompts, Turkish / English / code, teacher-forced):

| prior 0.5 only in layers | KL |
| --- | --- |
| 0-11 | 0.111 |
| 12-23 | 0.072 |
| 24-35 | 0.069 |
| 36-47 | 0.057 |
| all (no map) | 0.141 |

## Measured

RTX 5070 Ti Laptop 12 GB, Core Ultra 7 255HX, Swift 1.5 IQ3_XXS (GSQ-RCO), `STRATA_ROUTE_PRIOR=0.5`, decode speed from
agent turns in mirrored order (engine generation tok/s):

| map | KL all / code | decode tok/s |
| --- | --- | --- |
| none (every layer 0.5) | 0.141 / 0.198 | 113.9 |
| `000000000000` + 36 x `5` (off in layers 0-11) | **0.108 / 0.147** | 113.0 (-0.7%, run-to-run spread is ~3%) |
| layers 12-23 and 36-47 only | 0.095 / 0.128 | 108.1 (-5.0%) |

KL is against the same build and config with no prior (every change here has a floor of ~0.04-0.05 from the other
opt-in switches of the config, so the bands do not add up).

So the first map keeps the prior's speed with about a quarter less KL (a third less in code). In daily use since 12 October
(`STRATA_ROUTE_PRIOR=0.5`, `STRATA_ROUTE_PRIOR_MAP=000000000000555555555555555555555555555555555555`). Built and
smoke-tested on today's `main` (a0116b27). `docs/DETAILS.md` gets the table.

Sur le site

Liens install, modèles, releases.