Strata 0.1.45 最新

0.1.45: Qwen3.6-35B-A3B reads pictures, serves several requests at once and runs up to 40% faster; switches for multi-user servers; Mac and setup fixes. Answers of the Qwen3.8-Flash-Next models are the same as in 0.1.44.1 (byte-identical in our identity tests on every GPU machine we tested: CUDA sm_60 / sm_86, HIP gfx1201, Intel A750, Mac M4). Update with UPDATE.bat (Linux: ./update.sh); setup replaces the engine with 0.1.45.

Qwen3.6-35B-A3B

• Pictures. Setup asks "Do you want images?" for Qwen3.6, downloads the pinned mmproj (0.9 GB) and writes --vision. It is the same projector family as Flash-Next's, so the same image encoder reads it. Checked against llama.cpp on the same pictures, same prompt ids, teacher-forced on llama.cpp's own answer: first token the same in 5 of 6 cases (the sixth, two pictures in one request, differs at a near tie), mean KL 0.0005 to 0.035 against 0.002 to 0.056 between llama.cpp's own CPU and GPU runs. Answers on the test pictures are the same text llama.cpp gives.
• Several requests at once (--parallel N at setup, "parallel": N in the config, up to 8). Every window carries one token of each conversation, so the dense weights and every expert two conversations share are read once. Every slot decodes the tokens it decodes alone (10 prompts x 128 greedy, 2 / 4 / 8 slots, slots reused, a 2.5K-token prompt read beside decoding slots). An R9700 32 GB with every expert in VRAM, 128 new tokens per request, five interleaved pairs against llama.cpp's llama-server -np N -fa on: 1 / 2 / 4 / 8 requests 83.9 / 173.5 / 252 / 325 tok/s against 73.8 / 107 / 165 / 208 (1.14x to 1.61x, 5 of 5 pairs each). --batch-mtp (opt-in) lets every slot verify draft tokens: one request 1.6x, two 1.2x, four 1.1x.
• Setup offers it, with the price. Where every expert stays in VRAM, setup suggests up to 3-4 slots. On a card that does not hold every expert the slots take VRAM from the expert cache: a request alone runs 4-11% slower, and next to one request at a time WITH the draft layer, N together are only 1.0-1.15x faster (RTX 3060 12 GB at 8K context, 4 requests: 1.15x, alone -4%; RTX A4000 / Tesla P100 16 GB at 32K, 2 requests: 1.0x, alone -7 to -11%). Setup says that, offers the count the card can spare at the chosen context (12 GB: 4 slots at 8K, none at 32K; 16 GB: 2 at 32K), asks "How many requests at once?" (Enter keeps one at a time) and keeps any number you give. It is worth it for several users or agents at once, not for one chat.
• Faster. New decode kernels with the old kernels' bits (checked on the real activations: 0 mismatches in 18,360 comparisons), a prompt delta rule that loads one token ahead, and a bigger prompt chunk by default (4096 / 2048 / 1024 tokens, whatever fits; --prefill 512 brings the old one back). R9700 32 GB, UD-Q4_K_XL: decode 83.2 to 117.8 tok/s (1.42x, 5 of 5 pairs; llama.cpp 82), 4K prompt 3,453 to 6,220 tok/s (1.80x), with --mtp 2 167.7 to 183.7. UD-Q6_K_XL: decode 80.4 to 106.9. RTX 3060 12 GB (CPU-bound in the hybrid mode): decode +2%, prompt +3%. Text from the old chunk size differs a little on long prompts (mean KL 0.004 against chunk 512, the same size as the llama.cpp CPU-GPU floor); prompts shorter than the chunk are unchanged.
• Checked on: R9700 32 GB (UD-Q4_K_XL, UD-Q6_K_XL, Q8_0), RTX 3060 12 GB, Tesla P100 16 GB and RTX A4000 16 GB (UD-Q4_K_XL and UD-Q6_K_XL), with --batch 2 / 4 / 8, --batch-mtp, sampled requests and pictures: every slot equals its run alone. Not run: Windows hardware (the Windows zips contain the engine and start; Qwen3.6 on Windows AMD got a fix, see below, but is untested), Intel Arc (SYCL still refuses the family), the RTX 5070.
• A start whose buffers do not fit now says so with the numbers instead of a bare allocation failure.

Multi-user servers: opt-in switches (#2364, thanks havok2-htwo)

Everything here is opt-in; defaults are unchanged. For a server with several cards and many users:

• --batch above 8 on a layer split (16 slots on 4 cards = 4 groups of 4).
• STRATA_SLOT_COPY_DIRECT=1 and STRATA_SLOT_COPY_BATCHSYNC=1 (admission copy device to device), STRATA_CKPT_POOL=1, STRATA_CKPT_THREADS=1 (checkpoints), STRATA_ADMIT_MERGE_FIRST=1, STRATA_PIPE_BYIELD_NODRAIN=1, STRATA_BATCH_SELF_COMMIT=1, STRATA_QSA_SLOTS=1, STRATA_HANDOFF_PIPE=P, the server's STRATA_ADMIT_SHORT_FIRST=<tokens>, and STRATA_TRACE for admission phases.
• CUDA only, a build option: -DSTRATA_MMQ_TAB=ON with STRATA_MMQ_DIRECT=1 and STRATA_MMQ_GROUP_SORT=1 (pointer-table MMQ, resident experts read in place). It is off in the release builds and costs 36-38 MiB per GPU.
• Measured on 4x R9700 (HIP, IQ3_S with every expert in VRAM, layer split, three interleaved rounds): 16 users 207 to 324 tok/s (1.56x, 3 of 3 rounds), time to first token p50 20.6 s to 2.0 s; 8 users 1.20x; 16 users with 1.2K-token prompts 1.41x; one user unchanged. The slot count alone gives about 40% of the 16-user gain, the copy and checkpoint switches the rest. Every slot of a 16 and a 32 slot batch decodes the tokens it decodes alone. On small cards (2 cards, 16 GB) the gain is smaller and the memory cost real; see docs/BATCHING.md.
• `STRATA_SLOT_CKPT_MOVE=1` is experimental: do not use it. With it, tools/batch_groups_cache_test.py fails (the slot reuse pattern changes); all other switches pass it.

Faster, from your pull requests

We went through every open pull request with a speed claim, built each on the cards we have and ran the answer check against 0.1.44.1 before changing a default. What is on by default now gives the same answers as before (10 prompts and 4K / 20K exact prompt checks, identical on RTX 3060, Tesla P100, R9700 and Arc A750); each has an environment switch to turn it off. Everything that changes a number or is not faster everywhere is opt-in.

• Long prompts on AMD (#1749 vlupilin, #1943 2jztricks). The top-k kernel for very long contexts and an exact scorer for long prefill: on an R9700 a 131K-token prompt reads 8.7% faster, 230K 16%, 500K 79%; answers identical.
• Intel Arc (#2340 leadgtr7, #2207 aslater3). Q6_K decode no longer limited by ALU work and wide kernels in large-register mode: on an Arc Pro B70 decode is 3.5% (short prompt) to 4.2% (4K prompt) faster, same answers; the A750 kernel timings improve 1.7x at 4 columns.
• A resident-RAM copy keeps the prompt path's lend region (#1896 elightcap). With room in the RAM budget, decode went 7% and prompt reading 2.5% faster on an RTX 3060 (5 of 5 pairs).
• NVIDIA prefill and CPU (#1368 sergqwer, #1107 brenoperucchi, #2016 libratechw, #1949 maxious). Small, each 0.6-1.4% on an RTX 3060 / RTX A4000, and the AVX2 dot kernels keep their accumulators in registers (2-9% on the router micro-benchmark).
• Server (#1542, #1557, #1852, #1891, #1163, #1164). Tool-call and logit-bias parsing, prompt tokenisation from the last shared prefix and conversation-cache copies cost fewer milliseconds per request; no engine change.
• Opt-in, measured: STRATA_HIP_WMMA=1 on an R9700 (+44% prompt, +3.7% decode), the five decode switches of the CUDA series (+7% at 30K context on RTX 3060 and A4000), STRATA_EXCHANGE_ROTATE=1 (+5% decode on a 3060 with the resident RAM copy; do not use it with the resident copy under --expert-cache 1500 yet: the engine did not start on one 3060 box), STRATA_PCIE_BALANCE=1 (+5.6% on two A4000, slightly slower on single 3060 / P100), plus the Turing (sm_75) kernel series behind STRATA_SM75_PERF=1. We have no sm_75 card: those paths build, run bit-for-bit on a 3060 in emulation, and carry the author's speed numbers only. All switches are in docs/DETAILS.md and the backend pages.
• Not for you yet: STRATA_SIDE_MIXER=2 and STRATA_SH_STREAM=1 are slower on gfx1201 (0.41x and 0.44x decode); STRATA_QSA_WARP=1 crashes a Tesla P100. Leave them off.
• Moved to 0.1.46: the Volta/Pascal series. Together it made defau