#6442 [Engine] Inkling-2-Mini support (looped MoE from Thinking Machines)

open · @baptisterajaut · 12 评论 · 去 GitHub 看

Multi-GPUAMD / HIPNVIDIA / CUDAModels & quants

说明

### Area

[Engine]

### What do you want to do?

Run thinkingmachines/Inkling-2-Mini in Strata. Released today under Apache-2.0, 26.4B total / 3.4B active: the same weight class as Qwen3.6-35B-A3B, which Strata already serves, but it lands well above it. Its card puts it slightly ahead of the dense Qwen3.8-27B, itself clearly ahead of Qwen3.6-27B and Qwen3.6-35B-A3B. On a 24 GB card it would seem a natural fit for Strata's expert offload: the Q8_0 GGUF alone is 27.8 GB, before any long context.

From its config.json, the architecture would need real engine work: 24 layers where layers 6 to 17 run three passes with shared weights (48 layer applications per token), MoE with 128 experts, top-8 and no shared expert, 20 sliding-window layers (1024) and 4 global ones (5, 11, 17, 23), logit and MLP softcaps, and an MTP block drafter shipped in mtp.safetensors. The loop would also change the expert cache economics: the routers of layers 6 to 17 run on each of the three passes, so keeping those experts resident first and streaming the outer layers seems like the better trade.

The official checkpoint is NVFP4 + MXFP8 (19.5 GB). GGUFs already exist from ggml-org (BF16, Q8_0, IQ4_NL) and unsloth (UD-IQ1_S to UD-Q8_K_XL), built against llama.cpp PR ggml-org/llama.cpp#32694, still open; the vLLM one is vllm-project/vllm#69028.

### What do you do today instead?

Nothing yet, since neither llama.cpp nor vLLM support is merged. Happy to test a branch and report numbers on an RTX 3090 24 GB with 187 GB RAM (2x Xeon Gold 6252, two NUMA nodes), where we run Qwen3.8-Flash-Next Q6_K on 0.2.17 today.

本站相关内容

相关页面的快捷入口。