Strata 0.2.17

Downloads

Release notes

# Strata 0.2.17

The app and the Terminal are 0.2.17. The engines are 0.1.63 on every platform, and so is Control, the small program that installs and updates them. The update feed is sequence 129.

What changes for you

  • Update as usual (see Updating below). Your models, chats and settings stay.
  • The engines are new downloads: strata update and the app fetch them. Every platform has one (Windows, Linux, Mac, DGX Spark).
  • The route tail skip is off by default. It was on by default on NVIDIA and on Radeon RX 9000 cards, and answers differed slightly from a run without it. If you want it, turn it on (below).
  • Setup leaves 17 GB of RAM free, not 24, on a PC where the model does not fit in RAM. A 64 GB PC now holds 47 GiB of experts in RAM (it was 40). Details below.
  • The Coder model has a new pinned build (#3445).
  • On a PC with one NVIDIA card of 16 GB or less from the RTX 50 series, one more speed setting is on by default (below). It can be turned off.
  • New settings are off by default, unless the line for that setting says otherwise.

The route tail skip is off by default

  • What it is: "Skip rarely used parts of the model" in Expert settings (STRATA_ROUTE_TAIL_SKIP). When a verify window needs an expert that every token routes at rank 7 or lower, the engine neither copies nor computes it.
  • Why it changed: people reported quality loss, so it is off unless you turn it on.
  • What it changes in speed and in the text. On 2 x RTX A4000 (IQ3_XXS) decode with it Off is 47.0 tok/s in the first chat on two cards (50.8 on 0.2.16, where it was On) and 35.5 on one card (39.2). On an RTX 5090 (UD-Q4_K_XL, DFlash2, 5 kinds of request, 2 rounds) the mean decode is 73.4 tok/s Off and 94.5 On (x1.29); the forced-token KL against exact routing is 0.0126 with it On. Because of the new default, the BASE-05 answers of a Radeon R9700 (gfx1201) and a Tesla P100 (sm_60) differ from 0.2.16's in 10 of 10 prompts at the defaults and are identical with STRATA_ROUTE_TAIL_SKIP=7; on the RTX cards (sm_120) the two prompts that differ come from the --short-read default below, and are identical with --short-read 64.
  • Turn it on: Expert settings, "Skip rarely used parts of the model" = On (rank 7); in the Terminal, strata launch --set STRATA_ROUTE_TAIL_SKIP=7; or in the "env" of the model's run config. Ranks 1 to 9 stay available. A value you set yourself, even 0, is never changed.
  • Where Off comes from: Control sets STRATA_ROUTE_TAIL_SKIP=0 for every start unless you set the variable, and setup writes it into every Unsloth config, so engine 0.1.62 and older also run with it off under this version. Engine 0.1.63 itself reads an unset value as off on every backend.
  • If you updated from 0.2.16: you had "Automatic" or never touched it: it is now Off (the 0.2.16 "Automatic" was rank 7 on CUDA). If you had picked rank 7, or any other rank, that stays. A 0 stays Off. A PC with 64 GB in --resident-budget-gib mode used to run On with no entry and now runs Off.

Setup uses more of your RAM

  • The rule, for the Unsloth files UD-Q4_K_XL and UD-IQ4_XS: when their experts do not all fit in RAM, setup keeps RAM - 17 - the streamed KV cache GiB of them in RAM (0.2.16: RAM - 24). The other models (IQ2_XS, IQ3_XXS, IQ3_S, Q2_0) get no such budget, in 0.2.16 and 0.2.17 alike. A real 64 GiB limit, UD-Q4_K_XL, an RTX PRO 4500 Blackwell card:

| RAM budget | SSD read for a 65K prompt | decode | free RAM of 64 GiB | a later 8 GiB program | |---|---|---|---|---| | 40 (0.2.16) | 97.7 GB | 57.7 tok/s | 16.6 GiB | ran on | | 47 (0.2.17) | 37.0 GB | 84.2 tok/s | 9.3 GiB | ran on | | 52 | 0.9 GB | 83.4 to 88.6 tok/s | 6.6 GiB | killed |

  • Per PC size (UD-Q4_K_XL): 32 GB 8 -> 15, 48 GB 24 -> 31, 64 GB 40 -> 47. From 96 GB of RAM every expert is in RAM, as before. A PC with one shared memory for CPU and GPU (Strix Halo and the like) and the two-card layer split stay at 24 left.
  • A "64 GB" Windows PC reads 63.1 GiB. Setup now rounds the reading up to the PC's size, so it gets 47, like Linux.
  • "Leave 20% for my work" in the app keeps the old number (40 on 64 GB). The Terminal, strata serve, Docker and "Full performance" use the new one. A budget you set yourself, --ram-keep-gib and STRATA_RAM_KEEP_GIB win in every mode.
  • Installs from 0.2.16 move once, without a setup run, at the first strata command after the update, unless the budget is yours.
  • On Windows the engine takes at most free RAM at the start - 4.25 GiB, so a busy 64 GB PC can start with less than 47. Not measured on Windows.

Faster on NVIDIA Blackwell cards

Measured on an RTX 5090 32 GB (UD-Q4_K_XL, DFlash2) unless a line says otherwise, with identical answers: 27 of 27 BASE-05 prompts for each change.

  • Short prompts. On compute capability 12.0 cards the server reads prompts of up to 16 tokens through the decode windows and longer ones in one batch (it was 64). 24 tokens 172 -> 135 ms, 32 tokens 190 -> 140 ms, 64 tokens 403 -> 175 ms in the sweep. 2 of 9 BASE-05 prompts changed their text, KL 0.035 against the windows read. Back to the old rule: --short-read 64, or STRATA_SM120_PERF=0.
  • Long prompts. The hyper-connection up-mix kernel is on for 12.0 cards: a 32,768-token prompt 4,277 -> 4,160 ms (0.973), 8K and 16K level.
  • Decode, three switches on for 12.0 cards: the L2 prefetch (+1.4%), the Q8_0 one-row kernel (+1.2%) and the expert mode 8 (+1.8%), together +2.0% (20 of 25 paired runs up). =0 turns each off.
  • The CPU expert pool runs each expert's down rows as soon as its gate and up rows are done (STRATA_POOL_DEP): window -2.9% on a 22-worker EPYC box, same answers (27 of 27 prompts identical). STRATA_POOL_DEP=0 is the way back.
  • Two cards (layer split). The CPU prefill share is armed by default on a split of two CUDA cards, as it is on one card: prompts of 32 to 2,048 tokens take 0.39 to 0.61 of the time (84 tokens 1,193 -> 489 ms on two RTX 5060 Ti), decode unchanged (0.93 to 1.02). STRATA_SPLIT_CPU_SHARE=0 turns it off. The text of a two-card run differs from the run with it off in 5 of 9 BASE-05 prompts (2 x RTX A4000, IQ3_XXS); forced-token KL against the FP16 path 0.961 [0.899, 1.032] of the figure with it off.
  • DGX Spark (GB10). Two of the 12.0 switches are on for compute capability 12.1: decode +1.3% (geometric mean of 4 kinds, 5 paired rounds), prompts level.
  • Cards of 16 GB or less, compute capability 12.x (RTX 5060 Ti, 5070 Ti, 5080 and the like). STRATA_ADAPT_FETCH=2 is on by default (one card). On an RTX 5060 Ti 16 GB with the 0.1.63 engine (UD-Q4_K_XL, DFlash2, 3 rounds) decode is 1.02 to 1.23 of the figure with it off by kind of request (a 4,096-token prompt 35.5 -> 40.7 tok/s, a tool call 51.1 -> 62.6). The text is not the same bit for bit: 9 of 48 prompt rounds gave the same tokens; forced-token KL against the FP16 path 0.948 [0.830, 1.063] of the default placement. STRATA_ADAPT_FETCH=0 turns it off. It runs in the arena mode (the model fits in RAM); in the resident RAM mode (--resident-budget-gib, 28 in our test) the engine does not use it and says so at start-up. The start-up line for the sm_120 switches names it.
  • Other results, on by request: STRATA_ARENA_THP=1 decode +7.1% on that box (the first start took 12.5 minutes to fill the arena there); STRATA_PREFILL_CPU_SHARE_NT=32 with STRATA_PREFILL_CPU_SHARE_MAX=8192 on one RTX 5090 on PCIe 4.0 x8: a 4,096-token prompt 4,272 -> 1,828 ms, 1,024 tokens 1,292 -> 956 ms; slower from 16K up.
  • The CPU prompt share follows the card's PCIe link (compute capability 12.0 cards). The share keeps its limit of 3,072 tokens and its experts of 8 tokens on a fast link (RTX 5090 on PCIe 5.0 x16, 53.6 GB/s: nothing changes). On a slower link the engine reads the link, the experts the card does not hold and the CPU pool, and raises both: at 28.3 GB/s 6,144 tokens and 32-token experts, at 14.1 GB/s 12,288 and 64, at 3

Full notes on GitHub

Newer release: v0.2.18Older release: v0.2.16