Strata 0.2.2 Mais recente

0.2.2: Gyro on Intel Arc cards and on Macs, faster Gyro on NVIDIA cards, a Main / Helper / Off choice for each graphics card, one models folder for the app and the Terminal, an engine package for the RX 6000 series, an optional API key, and one definition of answer speed. It also fixes the issues listed below. Each speed claim says what it was measured on, and each new default says whether the answers stay identical. The engine packages are version 0.1.47 (0.2.1 had 0.1.46) and the app is 0.2.2. Update with strata update (Terminal), Settings > Update (app), or UPDATE.bat / ./update.sh (source installs).

What is new

• Graphics cards can be Main, Helper or Off. In the app, strata launch, setup, the server and Docker, each card gets a role. Main holds the last layers, the draft layer and the final check of every window; a Helper holds a share of the layers and of the expert cache; an Off card is not used and gets no process. The recommended card starts as Main, the others as Helpers, and a card with 8 GB or less starts Off. Short warnings, never refusals, appear for built-in graphics as Main, a card the engine cannot run, too little memory for the chosen setup, and no card at all. A CUDA, HIP or SYCL engine needs a card to start, so with every card Off the server stops in 3 seconds with one plain line; the Mac engine can run without one. Command line: strata launch --gpu-roles 1=main,0=helper (cards you do not name are Off), the same flag in setup, GPU_ROLES in Docker. On two RTX A4000 the roles gave the expected order (head and draft layer on the Main card, picture reader on the first card), an Off card stayed at 1 MiB with no process, and all-Off stopped as described. Two equal cards now put the lower number on Main, so the engine order is [1, 0] where it was [0, 1]; the cards run at the same speed, so no change in speed is expected. Not tested: the Docker flag, and AMD or Intel cards with roles. Same work: strata launch kept replacing the card you picked at install ("gpu": 1) with its own pick of GPU 0, which is fixed; the engine log's "GPU 0 / CUDA0" for the first card it is given is now explained by one line from the server; an old AMD hip_ordinal no longer overrules the card you name now; and a card list such as [1, 0] is passed to the image generator as 1,0.
• Gyro runs on Intel Arc (Linux, tested on an Arc A750 only). The SYCL engine has the trellis kernels (TQ2_T, TQK6, TQK7) and the Hadamard rotation, so Gyro-S and Gyro-M load and answer on an Arc card. Setup lists them like for any card, warns and never refuses when VRAM and RAM together are short, and keeps Gyro's n-gram table on the SSD. On the A750 (8 GB, Ryzen 5 5600X, 62 GB of RAM) Gyro-S reads a 1,406-token prompt at 82 tok/s (40 before the matrix-unit launch took the trellis types, 5 of 5 pairs faster), a 20,000-token prompt at 195 tok/s, and writes 15.8 tok/s with the MTP draft. The card holds 410 of 24,576 experts, so the CPU and the PCIe link carry the decode; for these packs the engine gives the link a larger share (0.48 instead of 0.29, +16% decode, 5 of 5 pairs). On real tensors of both files the kernels equal a host replay of their own arithmetic in every bit, and 7 of 10 test prompts give the same 48 tokens as the CUDA engine (the other three agree for 5 to 9 tokens and then differ at a near tie; first-token KL 1e-4 to 1e-2, top-1 always the same). Details: GYRO.md.
• A model that is slightly too big gets smaller settings by itself. When a card is short by a little, setup and strata launch now try a 32K, 16K and 8K context, a smaller prompt chunk and a smaller reserve, in that order, and keep anything you chose yourself. Gyro-M on the 8 GB A750 used to stop with no room for a single cached expert; it now starts with a 16K context, reserve 100 and --prefill 1024 (the estimate was 300 expert slots and the engine got 309), and it answered correctly (Canberra, 391) at 16.6 tok/s. A model that fits keeps its settings. The check is wired into the Intel Arc setup and the Terminal's launch; the app's own fit check is not changed.
• Gyro-S and Gyro-M run on Apple Silicon Macs. The Metal engine now has kernels for agentionai's trellis experts (TQ2_T, TQK6, TQK7) and the Hadamard rotation they need, in the decode window and in both prompt paths, and reads the published files unchanged. Setup, the app and strata launch list Gyro on a Mac next to the other sizes, never as the default. Gyro-S is 58.5 GB on disk, of which 28.8 GB is the n-gram table that is read from the SSD; its weights need about 38 GB to stay in memory, so a 48 GB Mac holds it and a 32 GB Mac streams the experts. Measured on a Mac mini M4 with 32 GB: 9.1 tok/s decode (5 interleaved pairs) and 109 tok/s on a 4K prompt. The Gyro-specific dense-weight tiles gave +5% decode over the generic ones (5 of 5 pairs, same tokens). Checked: the decoder equals the format's definition bit for bit on the real tensors, the rotation equals the CPU's bit for bit, and the expert kernels match a double-precision reference to 1e-7 on real experts of Gyro-S and of Gyro-M (read by range requests, since it does not fit this Mac). KL against the CUDA engine on the same file (4,995 positions of the quant-quality set, mean KL / top-1): Mac against CUDA GPU experts 0.0207 / 95.5%, Mac against CUDA CPU experts 0.0187 / 95.4%, and the two CUDA runs against each other 0.0186 / 95.8%: the Mac sits at the CUDA engine's own noise level. Not measured: Gyro-M end to end, a Mac with 48 GB or more, and the experts' streaming copy for Gyro (it needs 26 GB more disk than the test Mac had). It needs the 0.2.2 Mac engine; an older one refuses the file when it loads, and setup says so.
• Gyro decodes faster on NVIDIA cards. The gate/up and down kernels for agentionai's trellis experts now keep the codebook in shared memory and take four rows per warp. On an RTX 3060 (12 GB) Gyro-S went from 54.3 to 62.5 tokens per second (+15%) and Gyro-M from 50.2 to 54.8 (+9%), 5 interleaved pairs each, faster in all of them; prompts are unchanged. The answers are bit for bit the old ones (identity gate on Gyro-S, Gyro-M, Q2_0 and IQ3_XXS), and Q2_0 and IQ3_XXS are not touched. Gyro-S is now about 7% and Gyro-M about 16% behind Q2_0 on that card (16% and 23% before); STRATA_IQ_STAGE_GRID=0 goes back to the old lookup. Cards of other generations were not measured.
• An engine package for the RX 6000 series (built, not tested on a real RX 6000 card). Linux has rx6000 and Windows has rdna2, each with GPU code for gfx1030 and gfx1031 (RX 6800, 6900 and 6950 series, RX 6700 XT). They are offered to a PC whose card reports one of those two and to no other: the RX 7000, RX 9000 and Strix Halo packages are unchanged, and a PC with an RX 7000 or 9000 card next to an RX 6000 card keeps the package it had. The engine runs plain kernels on these cards (RDNA2 has no matrix cores) and plain hipBLAS (hipBLASLt ships no kernels for gfx1030, so there is no tuning table). Setup and the app say once that this package was built and not tested on a real RX 6000 card. What was checked: the Linux package builds, every program in it has code for exactly gfx1030 and gfx1031, the rocBLAS kernel files for both are in it, every library it imports is in the package or belongs to the system, strata --version runs, and the whole engine (all targets, tests included) compiles for both. What was not: a model on an RX 6000 card, on either system, and the Windows package itself (the Windows build scripts take the new architecture list, but no Windows rdna2 package has been built). Earlier reports on these cards came from people who built the engine from source (docs/AMD_HIP.md). If something goes wrong, open an issue with the end of strata-<model>.log.
• **The CPU expert pool splits per NUMA node on a machine with two NUMA nodes (from the branch behind #3201, not