Pull requests / #2769
#2769 Muse Spark: opt-in streaming reference runtime and CPU/CUDA parity
open · draft · @BColsey · 0 comentários · No GitHub
Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descrição
## Summary Refs #2762. **Draft: this is an experimental Muse Spark text reference runtime, not complete installer/API integration or verified 816B inference.** Adds an OFF-by-default `STRATA_ENABLE_MUSE` build option and a separate `strata-muse` executable. It reads standard `muse-spark` GGUF shards, keeps weights file-backed and evaluates decoded row tiles on CPU or through an optional CUDA/cuBLAS adapter. No model weights are included or downloaded automatically. ## What changed - Complete Muse text forward graph: gated local/global GQA, local NeoX RoPE, folded norm/residual scales, full-stream sigmoid routing with selection-only correction bias, latent projections, per-expert post-normalization before weighted reduction, and final logit scaling/softcap. - Dynamic geometry, shard/tensor/encoding checks, checked KV/scratch sizing, strict token-ID CLI, metadata EOS/EOT and streamed JSON logits. - Bounded CUDA matrix-vector adapter: explicitly allocates two 4 MiB buffers plus 256 output bytes; CUDA/cuBLAS overhead is additional. It is **not** Strata's optimized expert cache or a native GPU quantized kernel. - Independent NumPy FP32 oracle, deterministic synthetic GGUF, strict comparator and mutation tests that expose omitted/reordered graph operations. - Discoverable build/run documentation in `docs/MUSE_SPARK.md`. Existing Qwen numerical sources, model families, setup defaults, server/frontend, requirements and dependency pins are unchanged. The normal `strata --families` inventory does not advertise Muse. ## Actual validation Linux x86-64; Ryzen 9 3900X; CUDA 13.0; RTX 3060 12 GiB. The existing Qwen service was left running and remained healthy. | Check | Result | |---|---| | Integrated CPU build and focused CTests | 5/5 passed | | CUDA-enabled build and focused CTests | 5/5 passed | | Oracle unit tests | 13/13 passed, including 44 graph/router mutation cases | | Focused existing Python regressions plus oracle on GPU host | 114 tests passed | | CPU versus independent oracle, all 7 × 64 logits | max absolute error `1.25150586e-6` | | **Actual RTX 3060 CUDA execution** versus oracle, all 7 × 64 logits | max absolute error `1.19227037e-6` | | Comparator bound | `2e-5 + 2e-5 * abs(reference)` | | Muse-OFF Qwen reference build | binary SHA-256 unchanged from the same-day pre-change build | | Worker sanitizer checks | ASan/UBSan plus leak detection passed | The tiny test includes two layers, four experts/top2, local-window crossing, nonuniform/nonzero parameters and every Muse-specific graph operation. Separate native tests cover synthetic F32/F16/Q8_0 and production-sized IQ1_S/Q6_K rows, malformed storage/geometry, generation/EOS and allocation failures. Synthetic quantized-row tests are **not** published-checkpoint validation. Baseline caveat: on untouched upstream `296b59d2f73c8f04a805df29c2fa97655adace78`, the existing setup-golden suite has eight Unsloth subtest failures. Its goldens were not regenerated or changed. The 114 passing tests above exclude that known-failing suite. ## Outstanding / not claimed - No full 816B checkpoint download/run, model-quality evaluation, production memory measurement or usable-speed claim. - No tokenizer/chat-template integration, installer entry, OpenAI/Anthropic server integration, tool/reasoning parser, multimodal input, MTP or multi-GPU scheduling. - No HIP, SYCL or Windows build/device validation. CPU and CUDA are the implemented reference paths. - No global llama.cpp pin changes. Unsupported row encodings and scaled RoPE fail explicitly. - Mapped-file/page-cache residency, headers, allocator and library overhead are outside the calculated KV/scratch bound. Input shards must remain unchanged during execution; a failed write/run can leave partial JSON. This establishes a reproducible, tested numerical path toward #2762; it intentionally does **not** close the feature request or claim a drop-in Muse provider. AI-assisted implementation used isolated swarm workers and separate fresh-context reviewers. The coordinator inspected changes, integrated ancestry-preserving lane commits, reran tests and performed the actual GPU checks. All testing used isolated temporary containers; no existing model/service/driver/client configuration was replaced.
No site
Links install, modelos, releases.