#6506 [Engine] add Inkling 2 model family and dual-GPU looped MoE engine support
open · @cobra91 · 0 comentarios · En GitHub
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descripción
### Problem and result: Strata currently supports Qwen architectures (`qwen4exp`, `qwen35moe`), but has no native support for Thinking Machines Lab's Inkling 2 architecture (`inkling2`, 26.4B physical / ~5.4B active looped MoE with NVFP4 / Q8_0 weights and o200k tokenization). This PR adds complete end-to-end support for the Inkling 2 model family in Strata across the engine, control, CLI, and desktop UI. --- ### Upstream Lineage & Individual Contributions: Huge thanks to everyone across both Project Maya and Strata whose work and feedback made this integration possible: - **Peasant Smith & Project Maya Contributors**: Foundational architecture, build framework, and runtime design of [Project Maya](https://github.com/mw00/project-maya). - **OscarSheng (maya #974)**: Initial `ik2` engine architecture and kernel scaffolding. - **alesha-pro (maya #1095 / #981 / #1079)**: Core `ik2` engine completion, NVFP4/MMA kernels, and speculative decoding. - **@cobra91**: Windows MSVC build support, CUDA 13.x compatibility, and Dual-GPU Expert Parallelism (`IK2_EP=1` / `IK2_EP_SPLIT`) originally contributed in [alesha-pro/project-maya#3](https://github.com/alesha-pro/project-maya/pull/3) and merged into Maya PR #1095; full Strata integration across CLI, native engine bridge, Desktop UI, chat server, and o200k tokenizer. - **@mercant1982**: Setup contract analysis and bounded CPU fixture testing ([fixture gist](https://gist.github.com/mercant1982/3753b55578edf89b805504c0d6516e18)), identifying and verifying the text-only installation regressions (`KeyError: mmproj` and `KeyError: mtp`). - **@baptisterajaut**: Comprehensive benchmark validation on RTX 3090 24 GB (confirming 120–124 tok/s decode with bit-identical logits against reference), and identifying the `--prefill auto` chunking and `--vram-reserve-mib` handling. - **@merbanan**: Turing architecture compatibility testing and feedback on additional quant formats. - **@Algo-Enthusiast**: In-depth analysis of DSpark MTP speculative decoding on Inkling 2 Mini (looped MoE, 48 visits). - **@mrexodia (Duncan Ogilvie)**: Upstream Project Maya PR [mw00/project-maya#1106](https://github.com/mw00/project-maya/pull/1106) proposing Windows MSVC build, DRAM session cache, and minimal thinking fix. --- ### Licensing & Attribution: - All original MIT copyright notices are fully preserved across all `engine/src/ik2/` files and test suites: `Copyright (c) 2026 Peasant Smith, OscarSheng, alesha-pro, cobra91 and the Project Maya contributors`. - Original ggml notice preserved in `ik2_iq_tables.h`. - Resolves: https://github.com/Niko1221/Strata/issues/6442 --- ### What this PR includes: - **Dedicated CUDA engine** in `engine/src/ik2/` supporting looped MoE execution (12-layer core repeated across passes), sliding-window attention, relative bias, sinks, L2 clipping, and FP8/FP16 KV cache. - **Dual-GPU Expert Parallelism & Resident CUDA Graph** (`--gpus 2`, `IK2_EP=1`): Zero-copy mapped staging buffer (`g_ep_stage`, 648 KB) and multi-GPU resident CUDA Graph replay enabling ~80.7 tok/s decode on asymmetric PCIe dual-GPU rigs. - **Dual-GPU Speculative Decoding (`--draft`)**: Dynamic looped drafter support (4 taps / 3 layers for Mini, 8 taps for Small), multi-token Tensor Core MMA verify windows ($T \le 8$), and peer expert reduction (`ep_combine_block`) achieving 62.8 tok/s with zero cache misses. - **Windows MSVC & CUDA 13.x compatibility**, with direct mmap reading enabled by default on Windows to avoid physical host memory allocation failures. - **CLI & Setup robustness**: - Fixed `KeyError: mmproj` by moving projector path resolution inside the vision guard in `terminal/setup.py`. - Added `"mtp": 0` entry to `OWN_PATH_FAMILIES["inkling"]`. - Handled `--prefill auto` gracefully in `ik2_engine.cu` (defaulting chunk to 2048 instead of 0). - Wired `--vram-reserve-mib` into `ik2_engine.cu` to respect user-specified VRAM headroom reservations. - **Validation suite**: Full kernel unit test suite ported from Maya (`engine/tests/cuda/ik2_kernels_test.cu`), Python forward-pass reference (`ref/inkling2.py`), comparison utility (`tools/ik2_compare.py`), and documentation (`docs/INKLING2.md`). - **Setup & Catalog**: Pinned SHA-256 and size verification in `setup.py` for `Inkling-2-Mini-UD-Q4_K_XL.gguf`, with explicit `drafter: none` setting. - **Serving & Frontend**: `strata_tokenizer` support for o200k (`inkling2`) pre-tokenization with `ignore_merges`, plus `InklingRewriter` / `InklingOutputParser` in `terminal/serve/frontend.py` to seamlessly parse Inkling's typed segments (`<|content_thinking|>`, `<|content_text|>`) into the Strata web chat UI, Desktop App, and `/v1/chat/completions` API. --- ### Validation & Benchmark Performance: - **Build**: Built with MSVC 19.44 + Ninja + CUDA 13.4 on Windows 11 (`sm_89`, `sm_120`), and GCC/Clang on Linux (`sm_86`). - **Single-GPU (RTX 3090 24 GB, verified by @baptisterajaut)**: - **120–124 tok/s** all-resident decode (bit-identical logits with Maya #1095 and KL 9e-4 against fp32 reference). - **~1.4× faster than llama.cpp** on looped MoE due to Strata's custom fused CUDA kernels (`mv_q8p`, `attn_res_rms_f16`, `experts_mma_kernel`) eliminating the per-layer CPU launch overhead across the 48 looped visits. - **Dual-GPU (RTX 5080 16 GB + RTX 4060 Ti 16 GB, verified by @cobra91)**: - **Resident Greedy Decode (`fast_token`)**: **80.7 tok/s** (up from 49.2 tok/s baseline) via dual-GPU resident CUDA graph and 648 KB zero-copy mapped EP staging with 0 cache misses. - **Speculative Decode (`--draft`)**: **62.8 tok/s** (3.62 accepted tokens/step on code, 54.2 ms verification across 48 looped visits, 0 cache misses) accelerated by multi-token Tensor Core MMA (`to_f16_limbs_tokens`, `ep_combine_block`). - **Setup Fixture (verified by @mercant1982)**: Passes the bounded CPU fixture check with 0 setup errors. - **Web UI & Desktop App**: Verified live interactive streaming in the official Strata Desktop App UI and server with separate native thought accordion and text generation.
En el sitio
Enlaces a install, modelos, releases.