Pull requests / #2515

#2515 setup: pin -DSTRATA_Q6K_EXPERTS=ON and -DSTRATA_MMQ_KQUANTS=ON in the CUDA and HIP engine builds (stale CMake cache silently drops the K-quant kernels)

open · @p3lusa · 0 commentaires · Sur GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

## Summary

`setup.py` compiles the CUDA engine without ever passing `-DSTRATA_Q6K_EXPERTS`, relying on the CMake `option()` default. That default was `OFF` when the option was introduced (`ac7db2e5`, 2026-10-05) and flipped to `ON` in `465cd61d` (2026-10-12). **CMake does not rewrite an option that is already in `CMakeCache.txt`**, so a `build/` folder first configured while the default was `OFF` keeps `STRATA_Q6K_EXPERTS:BOOL=OFF` forever — even after `git pull` brings the new `CMakeLists.txt` with the `ON` default.

`source_hash(ENGINE_SOURCES)` fingerprints the source files (so setup *does* recompile after the pull), but it does **not** cover CMake option values. The result: setup recompiles the engine, silently, with the Q6_K gate/up grouped expert kernels still disabled.

This PR pins `-DSTRATA_Q6K_EXPERTS=ON` explicitly in the CUDA engine build (`setup.py`) and in the Dockerfile build, so a stale cache entry can no longer override the default the build promises.

## The failure (exact startup error)

Starting the model `Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` (Unsloth UD-Q4_K_XL, whose routed experts are Q4_K gate/up + Q6_K down) through `./setup.sh --host 0.0.0.0 --port 6001 --setup` (choice 6 → 1), the engine started and immediately refused to load the model:

```
[strata] engine started: .../engine/strata --serve --native .../Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 8192 --prefill auto --mtp 2
strata-q36: blk.34.experts are Q4_K/Q6_K, which the native expert kernels do not take (build with -DSTRATA_Q6K_EXPERTS=ON for Q6_K)
```

The engine binary in `engine/` had been built with `STRATA_Q6K_EXPERTS=OFF`, so `K::native_expert_supported()` (`src/qwen35/q36_engine.cu:973`) rejected the Q6_K expert layout and the model never loaded. Retrying with smaller `--max-context` (131072 → 65536 → 32768 → 8192) did not help — it is not a memory issue, the kernels were simply absent from the binary.

## Root cause

1. `build/CMakeCache.txt` on the affected machine had `STRATA_Q6K_EXPERTS:BOOL=OFF`.
2. The repo checkout (`296b59d2`, 2026-10-14) **does** contain the `ON`-default commit `465cd61d`, and its `CMakeLists.txt:83` reads `option(STRATA_Q6K_EXPERTS ... ON)`.
3. But the `build/` folder predates that flip: it was configured when the default was still `OFF`. CMake keeps the cached value and ignores the new default on re-configure.
4. `setup.py`'s recompile trigger (`source_hash`) saw the changed sources and recompiled — but recompiled with the cached `OFF`, so the Q6_K kernels were never compiled in.

Verified with a minimal CMake reproduction:
- configure with `-DSTRATA_Q6K_EXPERTS=OFF` → cache `OFF`
- change the default in `CMakeLists.txt` to `ON`, re-configure **without** the flag → cache stays `OFF` (the bug)
- re-configure **with** `-DSTRATA_Q6K_EXPERTS=ON` → cache becomes `ON` (the fix)

## Is it hardware-specific?

**No.** It is a CMake cache-staleness issue, independent of GPU, architecture, or OS. The only requirements to hit it are:
- a `build/` folder configured before the default flipped to `ON` (i.e. before `465cd61d`), and
- a model whose routed experts use Q6_K (Unsloth UD-Q6_K_XL, and UD-Q4_K_XL files that put Q6_K on the down-projections, e.g. `blk.34` here).

The affected machine was an **RTX 3080 Ti (sm_86, 12 GB VRAM), 26 GB RAM, Linux (Ubuntu)**, running the model in Strata's hybrid-experts mode (51/256 experts per layer in VRAM, the rest in pinned host RAM). The hardware only determines *how* the model runs (hybrid experts on a 12 GB card); it does not cause the failure. Any platform with a stale `build/` cache and a Q6_K-expert model hits the same error.

## The patch

- `setup.py` (`build_engine`, CUDA path): add `-DSTRATA_Q6K_EXPERTS=ON` and `-DSTRATA_MMQ_KQUANTS=ON` to the engine `cmake_build` definitions. An explicit `-D` on the configure command line overrides the stale cache entry, so the engine always has the Q6_K gate/up kernels and the `strata_kq` K-quant MMQ library its default build promises. (`465cd61d` flipped both options OFF→ON the same way and CMake caches both the same way — qiuzan2001's review; pinning only one leaves the other stale.)
- `setup.py` (`build_engine_hip`): both pins on the HIP configure too. `build-hip` has the same stale-cache case — the `ON` default never reaches an existing cache. Safe for the gfx906/#1396 concern: `setup.py` never builds gfx906 (its AMD path is gfx1100/gfx1101/gfx1102/gfx1200/gfx1201), and both flags match the `main` defaults, so a clean HIP configure is byte-identical — the `-D` only overrides a stale entry.
- `Dockerfile`: same pins, for consistency (the Docker build starts from a clean cache, so it already gets `ON` from the default; the explicit flags document intent and are robust if the default ever changes).
- `tools/test_setup_choices.py` / `tools/test_setup_amd.py`: assert both flags reach the CUDA and HIP engine def lists, so a future edit that drops a pin fails a test.

## Verification

On the affected machine, after reconfiguring with `-DSTRATA_Q6K_EXPERTS=ON` and rebuilding, the engine binary gained the Q6_K grouped-kernel instances (`native_gu_kernel<14>` / `native_down_kernel<14>`; template instantiations went from 102 → 143), and the same startup line that previously failed now reaches:

```
strata-q36: prompt experts on the native grouped kernels
strata-q36: context 8192 cells, prompt chunk 2048, MTP block loaded, ...
strata-q36: warmed up in 15.5 s ...
READY 8192 stop
```

The `blk.34.experts ... do not take` error is gone.

Sur le site

Liens install, modèles, releases.