Pull requests / #2769

#2769 Muse Spark: opt-in streaming reference runtime and CPU/CUDA parity

open · draft · @BColsey · 0 comentários · No GitHub

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descrição

## Summary

Refs #2762. **Draft: this is an experimental Muse Spark text reference runtime, not complete installer/API integration or verified 816B inference.**

Adds an OFF-by-default `STRATA_ENABLE_MUSE` build option and a separate `strata-muse` executable. It reads standard `muse-spark` GGUF shards, keeps weights file-backed and evaluates decoded row tiles on CPU or through an optional CUDA/cuBLAS adapter. No model weights are included or downloaded automatically.

## What changed

- Complete Muse text forward graph: gated local/global GQA, local NeoX RoPE, folded norm/residual scales, full-stream sigmoid routing with selection-only correction bias, latent projections, per-expert post-normalization before weighted reduction, and final logit scaling/softcap.
- Dynamic geometry, shard/tensor/encoding checks, checked KV/scratch sizing, strict token-ID CLI, metadata EOS/EOT and streamed JSON logits.
- Bounded CUDA matrix-vector adapter: explicitly allocates two 4 MiB buffers plus 256 output bytes; CUDA/cuBLAS overhead is additional. It is **not** Strata's optimized expert cache or a native GPU quantized kernel.
- Independent NumPy FP32 oracle, deterministic synthetic GGUF, strict comparator and mutation tests that expose omitted/reordered graph operations.
- Discoverable build/run documentation in `docs/MUSE_SPARK.md`.

Existing Qwen numerical sources, model families, setup defaults, server/frontend, requirements and dependency pins are unchanged. The normal `strata --families` inventory does not advertise Muse.

## Actual validation

Linux x86-64; Ryzen 9 3900X; CUDA 13.0; RTX 3060 12 GiB. The existing Qwen service was left running and remained healthy.

| Check | Result |
|---|---|
| Integrated CPU build and focused CTests | 5/5 passed |
| CUDA-enabled build and focused CTests | 5/5 passed |
| Oracle unit tests | 13/13 passed, including 44 graph/router mutation cases |
| Focused existing Python regressions plus oracle on GPU host | 114 tests passed |
| CPU versus independent oracle, all 7 × 64 logits | max absolute error `1.25150586e-6` |
| **Actual RTX 3060 CUDA execution** versus oracle, all 7 × 64 logits | max absolute error `1.19227037e-6` |
| Comparator bound | `2e-5 + 2e-5 * abs(reference)` |
| Muse-OFF Qwen reference build | binary SHA-256 unchanged from the same-day pre-change build |
| Worker sanitizer checks | ASan/UBSan plus leak detection passed |

The tiny test includes two layers, four experts/top2, local-window crossing, nonuniform/nonzero parameters and every Muse-specific graph operation. Separate native tests cover synthetic F32/F16/Q8_0 and production-sized IQ1_S/Q6_K rows, malformed storage/geometry, generation/EOS and allocation failures. Synthetic quantized-row tests are **not** published-checkpoint validation.

Baseline caveat: on untouched upstream `296b59d2f73c8f04a805df29c2fa97655adace78`, the existing setup-golden suite has eight Unsloth subtest failures. Its goldens were not regenerated or changed. The 114 passing tests above exclude that known-failing suite.

## Outstanding / not claimed

- No full 816B checkpoint download/run, model-quality evaluation, production memory measurement or usable-speed claim.
- No tokenizer/chat-template integration, installer entry, OpenAI/Anthropic server integration, tool/reasoning parser, multimodal input, MTP or multi-GPU scheduling.
- No HIP, SYCL or Windows build/device validation. CPU and CUDA are the implemented reference paths.
- No global llama.cpp pin changes. Unsupported row encodings and scaled RoPE fail explicitly.
- Mapped-file/page-cache residency, headers, allocator and library overhead are outside the calculated KV/scratch bound. Input shards must remain unchanged during execution; a failed write/run can leave partial JSON.

This establishes a reproducible, tested numerical path toward #2762; it intentionally does **not** close the feature request or claim a drop-in Muse provider.

AI-assisted implementation used isolated swarm workers and separate fresh-context reviewers. The coordinator inspected changes, integrated ancestry-preserving lane commits, reran tests and performed the actual GPU checks. All testing used isolated temporary containers; no existing model/service/driver/client configuration was replaced.

No site

Links install, modelos, releases.