#6520 [Engine] Inkling-2-Mini from the official NVFP4/MXFP8 checkpoint, expert offload and CPU experts for 8-16 GB cards
open · @baptisterajaut · 0 comentários · No GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux
Descrição
**Problem and result**
Since 0.2.19, Strata runs Inkling-2-Mini (thinkingmachines/Inkling-2-Mini: 128-expert top-8 MoE, layers 6-17 looped three times, so 48 layer applications per token, plus a ViT for pictures) through `strata-inkling` and its SQ packs. Its setup also lists the official safetensors as a source, but `strata-inkling` reads them only through its EP backend, where attention runs on the host: on an RTX 3090, 2.6 tok/s resident and 2.3 tok/s at 8 GB, and a segfault with the default `--chunk 512` (48 works). This PR makes that source run `strata-ink2`, an engine that loads the official checkpoint as shipped (NVFP4 experts for layers 0-19, MXFP8 for 20-23, BF16 dense weights and vision tower), installed from `./setup.sh --family inkling-2-mini --model official` (or `--family inkling2`), and served by the Strata server with its chat template, thinking and tool-call markers, reasoning-effort words, pictures and a conversation cache.
This duplicates work from #6506 (cobra91), but I went the other path on purpose: the official checkpoint is already quantized enough to work on as is, so there is no requantization, and the quality is the reference's. Against the transformers implementation (huggingface/transformers#51050) in fp32, the token path with fp32 K/V is at mean KL 1.4e-12 on a 1,600-token text, top-1 identical on every position. With the default f16 K/V it is 1.3e-3 (router near-ties), against 5.4e-3 for ggml-org's IQ4_NL GGUF through llama.cpp on the same text and reference.
The PR is big because the goal was not a minimal engine strapped on in a hurry, but an integration that is already good where Strata matters, on 8-16 GB cards. On an RTX 3090 limited to 8 GB, with the experts that do not fit computed on the CPU, it decodes at 60-66 tok/s with a 32K context.
**Related discussion**
#6442, opened 2026-11-20 before any code. A public WIP branch was posted there at 11:05 UTC on 2026-11-21, more than three hours before #6506 was opened. Benchmarks of #6506 and maya against this branch are in my comments on #6506.
**Approach**
- **The engine.** `strata-ink2` is a standalone CUDA engine (`engine/src/inkling2/`). Decode is one captured graph per token. Prefill uses cuBLAS for the dense products and a tiled grouped GEMM for the experts, with weights decoded on the fly. `strata-ink2ref` is the plain reference path used to check it, and `terminal/tools/inkling2/` has the oracle and KL tools.
- **Expert offload.** `--vram-reserve-mib N` leaves N MiB of VRAM free, and the experts that do not fit live in pinned RAM. Placement uses decayed use counts with an admission margin, and promotions run in the background.
- `--cpu-threads N` computes the non-resident experts on the CPU: AVX-512 or AVX2 kernels, one thread pool per NUMA node, each expert's memory bound to its node. Doorbells in mapped memory, written from inside the decode graph, tell the pools what to compute, so there is no host sync per layer. With `--spec 1`, a verify block's missing experts go to the same pools (up to 8 tokens and 64 token-expert jobs per layer visit).
- `--kv-grow 1` (merbanan's) lets the K/V grow into the expert pools as a conversation grows. Each pool keeps 64 slots: a prompt shrinks its chunk near that floor, and one that cannot fit ends with a "context is full" error while the engine keeps serving.
- **Conversation cache.** `--conversation-cache-mib N` keeps conversations in pinned RAM. A checkpoint at each prompt end lets the next turn continue even when the client sends the history back without the model's reasoning (most agent clients do, and the server re-renders the history, so plain prefix reuse never fired after turn 1). Conversations set aside, an agent and its subagents taking turns, are taken up again, and a long shared beginning is read once.
- **Pictures.** The ViT and projector run in the engine in fp32, loaded into RAM at the first picture, so text-only use costs no VRAM. The server sends them as Hy4-Flash does (`image=` keys). The conversation cache keys picture tokens on the picture's content.
- **Speculative decode.** `--spec 1` uses the model's own DSpark drafter (`mtp.safetensors`). A draft token is kept only when it is the target's own choice (greedy) or passes the usual acceptance test (sampled), and it falls back to plain decode when that is faster. The verify step reads the drafts through other kernels than plain decode, with another summation order, so a near-tie can flip: on a 512-token chat answer, greedy output matches plain decode for 273 tokens, then parts where the top two logits are 0.2 apart. Plain decode is no different: reading the same prompt token by token instead of as one chunk changes that answer from token 55 on, at a 0.005 gap.
- **K/V formats.** f16 by default; i8, i8r, and merbanan's k8v8 / k8v4.
- **merbanan's work.** Ten of his patches are merged in with his authorship kept, each followed by the fix it needed beside the CPU experts, the placement and `--spec`: GGUF loading (IQ3_S / IQ4_NL / Q8_0), `--kv k8v4` / `k8v8`, `--kv-grow`, elastic prompt buffers, routed short prompts, a decode-once GGUF prompt GEMM, a chunk size that shrinks to fit, a smaller offload margin, and a Turing startup fix.
- **AMD.** jingx8885 ported both programs and setup to ROCm/HIP for wave32 RDNA cards (gfx10 to gfx12): a WMMA adapter for the tensor-core kernels, HIP virtual memory for the pools, and a HIP build in setup. Every change sits under the HIP build, the CUDA code is unchanged.
- **Setup.** The official source of the `inkling-2-mini` family, the SQ packs staying on `strata-inkling`: the checkpoint at a pinned revision with every file's size and SHA-256, the tokenizer pack, the `strata-ink2` build, and engine arguments sized from the card, the RAM and what a desktop already holds. Per card size, and what each config measures, is in `docs/INKLING2.md`.
Everything engine-side is opt-in or specific to `strata-ink2` / `strata-ink2ref`. Outside them: the server's Inkling family switch and image gate, a fix for thinking-off requests on Inkling (`enable_thinking: false` used to leave effort 0.9 in the prompt), and setup's CMake minimum raised to 3.25.2, because 3.24 and Debian 12's 3.25.1 cannot configure the engine's CUDA20 targets with nvcc 13; that last one is a separate commit and can be split out.
**Validation** (RTX 3090 24 GB, 2x Xeon Gold 6252, CUDA 13.2, Linux; engine and server at 2daf740df, and after merging 0.2.19 (f30718f41) the engine is bit-identical, `--spec` gives the same 128 tokens, and the server passes thinking, effort, tool-call, streaming and picture checks on 24 GB; the later commits add the docs page and the AMD port, whose CUDA build has the same SASS and PTX and gives bit-identical outputs; the picture and conversation-cache identity checks ran on their branches before integration)
- **Passed:**
- Against transformers in fp32 (1,600-token text): token path with fp32 K/V, mean KL 1.4e-12, top-1 100%; tensor-core prefill, 2.1e-7 with the routing fixed (free routing 1.2e-3, from router near-ties). Pictures: mean KL 1e-7 or less in fp32 on padded, transparent and multi-picture prompts, same greedy answers; the ViT's embeddings within 5e-5 of the reference.
- Kernel changes are checked against the previous build with logit dumps. This head is bit-identical to 6532963da on the token path (fp32 K/V) and the tensor-core prefill path; `--spec` gives the same 128 tokens as plain decode, resident and at 8 GB (longer answers can part at a near-tie, see above). `--spec 1 --cpu-threads 22` at 8 GB (PCIe share off) gives the same 512 tokens as `--spec 1` resident, its verify logits at KL 3.4e-13 from the resident ones on the exact path; the other paths stay bit-identical to f30718f41.
- Speeds with the configs setup writes (VRAM limited to the card size; 257-token chat prompt, 512 tokens; prompt speed at 16K):
- 24 GB: 88-90 tok/s (108 on a short prompt with `--spec`), prompt 2,380 tok/s.
- 12 GB with a desktop, 64K: 73-75 tok/s, prompt 717 tok/s.
- 8 GB, 32K: 66 tok/s headless (f16 K/V), 60-61 with a 500 MiB desktop (i8r), prompt about 565 tok/s.
- A 960x686 picture (432 tokens): first token in 0.63 s resident and 2.27 s at 8 GB; 1.27 s and 2.92 s for the first picture after start, which reads the ViT into RAM.
- Conversation cache: restoring a parked conversation gives bit-identical tokens and logits to one that never left (f16, i8, k8v4, `--spec`, `--kv-grow`, 8 GB with offload on the GPU only). With two agents taking turns and their reasoning dropped from the history (6 requests, prompts from 6K to 17K tokens), the four returning turns wait 8.6 s in all for their first token instead of 22.3 s resident, and 36-37 s instead of 100 s at 8 GB with a desktop.
- Setup: a fresh clone with a fresh HOME installs and starts with no manual step; through `./setup.sh`, the server passes its smoke checks on the 24 GB card and a simulated 8 GB one (thinking on/off, effort words, a tool call round trip, streaming, a 30K-token prompt, a picture).
- Tool calls, through BFCL v4 multi_turn_base on the 24 GB config (native function calling, temperature 1.0, effort medium): all 1,199 tool calls parsed with valid JSON arguments, none was left in the answer text, and the server logged no call found inside the thinking or failing to read as one; the 7 turns without a call are the model asking for a missing detail or declining. Two runs of the same 100 of the 200 tasks (even ids, 25 per domain, one sample each, same settings) scored 37% and 45%: at that size the score checks the chain, it does not rank the engine against published scores.
- `python3 -m pytest serve` and `tests/terminal` in `terminal/`: no new failures (see below).
- **Failed:** after merging 0.2.19, the same tests as on 0.2.19 itself (c6f6427d9) on this machine: 5 in `test_stop_closed_pipe.py`, and in `tests/terminal` the TUI modules that do not collect (no `textual` in the system Python).
- **Not run:**
- AMD: no AMD card here. The ROCm/HIP port is jingx8885's (baptisterajaut/Strata#16), measured by him on a Radeon 8060S (gfx1151): mean KL 4e-12 against the reference path, `--spec` matching plain decode, the server end to end. The targets are built on Linux HIP only and stay out of the default HIP build, so the other engines' HIP builds compile nothing new. SYCL: nothing new compiled.
- Windows, and real 8, 12 and 16 GB cards: the small cards were simulated on the 3090 by limiting its VRAM. 16 GB was computed, not run.
- Blackwell: native NVFP4 kernels would gain a lot there, but I am not fortunate enough to own such a GPU, so nothing here is tuned or tested on one.
**Limits**
- One GPU. `--cpu-threads` on Linux only, at most two NUMA nodes.
- With `--cpu-threads`, greedy output is not repeatable run to run: which misses go over PCIe depends on timings, and two runs can differ at mean KL about 1e-4. `INK2_PCIE=0` makes it repeatable.
- As with Hy4-Flash, the server counts a picture as one placeholder token in `usage.prompt_tokens`; the engine expands it (432 tokens for a 960x686 picture).
- `--spec` with `--cpu-threads` pays only when the card holds most experts: the drafter takes about 2 GiB of expert pool, and a verify computes 8 rows to keep about 3. Decode with and without it (chat prompt / short prompt, tok/s): 8 GB 66.9 → 51.8 / 64.6 → 59.9; 12 GB 80.6 → 72.3 / 81.5 → 87.9; 24 GB at 256K with i8r K/V and CPU experts 82.9 → 84.7 / 83.3 → 100.0. Setup adds it in offload only where the pools hold about 70% of the experts (in practice 24 GB cards at long context).
- GGUF models run without `--cpu-threads` and `--spec`, and setup does not install them. ggml-org's IQ4_NL file (Q8_0 in layers 20-23) loads, at mean KL 5.9e-3.
- Not offered in `strata launch` or the desktop app; `strata-inkling`'s own server path is unchanged, and its tool-call parser stays for it (strata-ink2 keeps its own, which keeps the reasoning).
**Next, not in this PR**
- Prefill: No site
Links install, modelos, releases.