Strata v0.1.42

About 13-16% faster decode on one NVIDIA GPU or a layer split, up to 30% more on a single card whose CPU outruns its PCIe link, a long list of fixes for multi-GPU splits, Windows AMD, Intel Arc and tight-RAM machines, and new opt-in speedups for AMD and multi-GPU cards. Two new NVIDIA defaults change answers a little (STRATA_ROUTE_TAIL_SKIP=7 and the measured PCIe share; each has a setting that turns it off). Update with UPDATE.bat (Linux: ./update.sh); setup replaces the engine with 0.1.42. The five fixes planned as 0.1.41.1 were never released and are part of this version.

Faster

• Skipping the tail of missed experts, on by default on CUDA (`STRATA_ROUTE_TAIL_SKIP=7`, from PR #1668 by LeGeRyChEeSe). In a verify window, an expert that is not in VRAM and that every token of the window ranks 7th or lower in its top 10 adds little to any token. It is now neither copied nor computed; the other experts keep the router's weights. Decode against =0, 5 interleaved rounds on fresh engines (story, code and a 6K-token document), median:

| Machine, model | Decode | Rounds faster | |---|---|---| | RTX 3060 12 GB, IQ3_XXS | +15.6% | 5/5 | | RTX A4000 16 GB, Q2_0 | +12.9% | 5/5 | | Tesla P100 16 GB, IQ3_XXS | +14.6% | 5/5 | | 2x RTX A4000, layer split (99.7% of experts resident) | -0.2% | no change |

A machine that holds nearly every expert gains nothing; prompt reading is untouched. It applies to CUDA serve with one GPU or a layer split, and is off when every expert is in VRAM, with --batch slots, with a peer or helper GPU, and on AMD and Intel. It changes answers slightly (quality numbers under "Answers can differ"); STRATA_ROUTE_TAIL_SKIP=0 restores them.

• The PCIe share is set from what your CPU measures (CUDA, one GPU). Missed experts are split between the GPU (read over PCIe) and the CPU; the default share came from a start-up link probe alone and ignored the CPU. With no --pcie-frac, the engine now times the CPU pool in the first decode windows (about 10 s) and sets the share to balance the two, in steps of 0.05 between 0.10 and 0.60, at most three times; each change prints one PCIe share: line. Decode against the old rule, STRATA_ROUTE_TAIL_SKIP=7 on in both, interleaved pairs, median:

| Machine, model | Share old -> new | Decode | Pairs faster | |---|---|---|---| | RTX 3060 12 GB, IQ3_XXS | 0.55 -> 0.20 | +30.0% (47.7 -> 62.1 tok/s) | 5/5 | | Tesla P100 16 GB, IQ3_XXS | 0.29 -> 0.20-0.25 | +1.1% | 4/5 | | RTX A4000 16 GB, Q2_0 | 0.30 -> 0.30 | no change | (same share) |

--pcie-frac N (and --calibrate, which writes it) always wins, and a request's own pcie_frac is kept. It applies to the native packs of the i-quants (Q2_0's own pack keeps its fixed share); off on a layer split, with --batch, a peer GPU, every expert in VRAM, and on AMD and Intel. A start on a busy CPU can raise the share; the cap is 0.60. STRATA_PCIE_FRAC_DEFAULT=old restores the 0.1.41 rule.

• gfx12 prompt switches on by default (#1478, report by jkuepker). Five prompt-path switches (STRATA_GDN_HEAD, GDN_PP=2, GDN_CONVL2, GDN_NOY, CVEC_FUSE) are on for gfx1200/gfx1201 cards such as the Radeon AI PRO R9700; residual and logits are byte-identical to the old path at 4K and 20K tokens. R9700, IQ3_S, 10 pairs: prompt +0.8% at 4K, +1.6% at 16K, +1.8% at 32K (the reporter saw +8% and +4% on another config). STRATA_GFX12_DEFAULTS=0 turns them off.

Final build against 0.1.41, default settings, 5 interleaved pairs, medians. The TAIL_SKIP gain comes on top of these for NVIDIA. A few percent on a shared box or from the draft acceptance rate is noise; no machine got consistently slower:

| Machine, model | Story decode | Code decode | Prompt 4K | |---|---|---|---| | RTX 3060, IQ3_XXS | +10.4% | +0.3% | +4.8% (shared box, 620-686 spread) | | Tesla P100, IQ3_XXS (start-up PCIe probe, default flags) | -3.6% | -2.1% | 0.0% | | Tesla P100, same with --pcie-frac 0.29 (8 pairs) | -0.5% | -0.1% | 0.0% | | RTX A4000, Q2_0, 1 GPU | +4.3% | -0.1% | 0.0% | | 2x RTX A4000, Q2_0, layer split | +3.7% | -0.2% | -0.1% | | R9700, IQ3_S (two boxes) | +0.03% / +0.1% | +0.01% / +0.15% | +0.9% / +0.3% (16K +1.5%) | | Arc Pro B70, IQ3_S, MTP | +0.9% | n/a | +0.1% | | Arc A750, IQ3_XXS, default / --pcie-frac 0 | -4.4% / +3.0% | n/a | -1.0% / equal in the fast runs |

On an RTX 5070 (Windows, Ryzen 5 7600) with Q2_0 and setup's own settings, 3 interleaved rounds: decode 77.0 -> 85.6 tok/s (+11.2%, 3 of 3 rounds faster, the tail skip's share), prompts at 4K 2,192 -> 2,193, 16K 2,778 -> 2,820, 32K 2,797 -> 2,833 tok/s.

The P100 and A750 default-flag swings come from the start-up PCIe probe (see "Answers can differ"); with the share fixed they are inside noise. Eight clients on 4x R9700: -0.7% total throughput, inside the run-to-run range, 0 errors.

Fixes

Crashes and regressions first.

• A card order you gave is kept (#1576, #1590, #1760). An automatic layer split puts the faster card last. On a pair with unequal VRAM that put the head, the draft layer and the verify on the card least able to hold them: the reporter's 12 GB card ended with a 3 GiB expert cache and decode fell from about 58 to 44 tok/s. The reorder is now skipped when it would put the last stage on a card with less total VRAM than your config chose, and cards whose speed scores are within 5% keep your config's order. Thanks to nekomario28.
• No one- or two-layer stages in the automatic split (#1616). A one-layer stage holds about 512 expert slots on a 12 GB RTX 3060 and clamps every card's prompt chunk. If the first pick has one, the engine searches again among placements with at least three layers per stage and prints both. Three is a rule of thumb from the reporter's numbers; we have no rig of that shape, so tell us if your split still looks odd. A near tie between two splits on a mixed pair now prints a warning (#1428).
• Windows AMD (HIP). A bare --prefill auto stops at 6,144 tokens (#1630): on an RX 7900 XTX the first long prompt at 8,192-token chunks charged about 70 GB of commit and the engine died (0xC0000409); it costs about 14% prompt speed there (645 against 748 tok/s). A value you pass is kept, with a warning from 8,192. Serve start prints the commit left and warns when planned host allocations plus 4 GiB do not fit, and the conversation cache uses the smaller of free RAM and free commit (#1607). The automatic expert cache keeps its 2,560 MiB VRAM reserve on 16 GB cards too (#1709). We have no Windows AMD box: these are guards that compile and pass their tests, not seen on a card.
• AMD, 2+ GPUs: the default checkpoint save of a 104K-token prompt no longer fails. On two R9700 the save failed with "operation would make the legacy stream depend on a capturing blocking stream"; the snapshot copies ran on the legacy stream while another thread captured a graph. They now use a private non-blocking stream. Final check, 104K tokens, 2 cards, defaults, 5 interleaved pairs: 0.1.41 failed 3 of 5 runs, 0.1.42 passed 5 of 5 with the same output every run.
• CPU prompt share on an RTX 5090 (a user report). We could not reproduce the crash, so the cause is not proven. The 0.1.41 default share is now skipped when free RAM is below its buffers plus 3 GiB, when the Windows commit left is below buffers plus 4 GiB, or when its experts are file pages RAM cannot hold; the log says why. An explicit STRATA_PREFILL_CPU_SHARE is kept and only warns. If the host cannot give the page-locked buffers, the share ends for that run instead of failing the chat. It is also not armed when every expert is in VRAM (#1595).
• End of the context (#1700, midagedev). A 4,094-token prompt with --max-context 4096 --max-new 2 --spec 4 stopped with "ran out of context"; it runs now.
• Intel. Without --mtp the engine could crash at exit (#1642): 6 of 6 runs on an Arc Pro B70, 0 of 3 now. After a batch