#5610 [Benchmark] Independent distributional quality: Strata vs llama.cpp vs BF16 truth, 399,600 teacher-forced positions
closed · @Andrey0710 · 1 コメント · GitHub で見る
BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationWindows
本文
Result (short)
I ran an independent, teacher-forced distributional comparison of Strata against llama.cpp
on the same GGUF files, with a full-precision (BF16) run of the same model as the ground
truth, on a ~400,000-position public-domain corpus. Bottom line:
**No distributional degradation was found.** On every quant where Strata was measured,
its agreement with the BF16 truth is equal to or better than the reference llama.cpp
build's on the same file:
| Quant | llama.cpp: top-1 / KL | Strata: top-1 / KL | Verdict |
| --- | --- | --- | --- |
| UD-Q4_K_XL | 94.02% / 0.0526 | **94.39% / 0.0461** | Strata closer to truth |
| UD-IQ4_XS | 92.29% / 0.0852 | **92.53% / 0.0808** | Strata closer to truth |
| IQ3_S | 89.80% / 0.1577 | 89.81% / 0.1580 | parity |
- **Quantization, not the engine, dominates the divergence.** The quant ladder
(llama.cpp vs BF16: KL 0.053 to 0.233 across Q4_K_XL to IQ3_XXS) is 2-4x larger than the
between-engine disagreement on the same file (KL 0.015-0.028, top-1 95.5-96.6%).
- Strata's engine is bit-deterministic in the fixed yardstick configuration (two identical
runs: KL 0.0000), so every measured divergence is real, not noise.
**Comparable with the project's published quality tables.** The measurements use the
project's own yardstick (`tools/quant_quality.py`, teacher-forced top-64), so the numbers
are directly comparable with the quality data in docs/UNSLOTH_Q4.md and docs/QUANTS.md -
with a new, larger corpus and a BF16 (not Q8_0) reference.
## Setup
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation 96 GB; RAM 253 GB; Windows, CUDA 13.3.
- Strata 0.2.12, engine 0.1.58 (release binary).
- llama.cpp: unsloth b12128-mix-2f23ac9 CUDA build ("the reference").
- Weights: unsloth UD-IQ4_XS (87.2 GB, 4 shards, **Dynamic V3.1**, shards 2026-11-04),
unsloth UD-Q4_K_XL (111.3 GB, **pre-V3.1** revision of 2026-08-29, downloaded 2026-10-21 - the repo's Q4_K_XL was re-packed on 2026-11-04 into the mix-n-match engram layout, so our copy is the older revision), ISTA-DASLab
GSQ-RCO IQ3_S (83.6 GB) and IQ3_XXS (75.8 GB), unsloth BF16 (345.7 GB, 8 shards).
GGUF metadata does not carry the unsloth quant generation - shard dates and the HF
release timeline are the provenance (files are updated in place).
- Corpus "standard-v2": 400 segments x 1,000 tokens, interleaved public sources -
llama.cpp/ggml sources (MIT), llama.cpp/Strata docs (MIT), Pride and Prejudice and
War and Peace (public domain). SHA256 of every source in the manifest.
## Method
Teacher-forced comparison on identical token id sequences. Per position each side writes its
top-64 log-probabilities (Strata: `STRATA_LOGPOS` / `STRATA_LOGPOS_TOPK=64` through the
project's own yardstick `tools/quant_quality.py run` with its fixed configuration - expert-cache auto, FP16 KV, no adaptation, no draft/speculation; llama.cpp: a small custom
dumper over `llama_decode`/`llama_get_logits_ith` writing the same `STRATA_LOGPOS` row
format). Metrics: top-1 argmax agreement, KL(REF||RUN) over REF's top-64 renormalized, KL p99,
and the text's perplexity under each side.
The BF16 ground truth ran through llama.cpp with the MoE experts and the per-layer
n-gram table streamed (mmap + lazy reads + tensor buffer-type overrides), ~33 s per 1,000
positions. Strata's engine was bit-identical across two identical runs (KL 0.0000), which
sets the noise floor at zero.
## What is attached
- Full report with all tables, the anatomy of divergences (near-tie share,
gap distributions, worst positions), the depth analysis, and the limitations: `REPORT-FULL.md`
- Raw distributions (TSV + sidecars) and all comparison JSONs - on request (~4 GB)
- Scripts: corpus builder, the llama.cpp reference dumper (C++), alignment and analysis
scripts, exact run commands
- Token ids of the public corpus + manifest with SHA256 of every source file
## Scope
Distributional comparison only (teacher-forced positions, fixed yardstick configuration,
single GPU, Windows). The full list of limitations is in the attached report.
## Notes
- The measured engine-vs-engine disagreement (KL 0.015-0.028) is 2-4x smaller than the
quantization cost itself. On the published corpus, 3.92% of positions diverge in argmax;
2.49% carry a gap >= 0.2 nat and 0.45% a gap >= 1.0 nat - so the earlier near-ties-only
reading of docs/UNSLOTH_Q4.md softens at scale, but the fidelity-to-truth outcome does
not change.
- Happy to share the full raw distributions, re-run anything on request, or adjust the
methodology if the maintainers see a flaw.
- Preparation note: the benchmark harness, the reference dumper and the analysis were
built and verified with an AI assistant (GLM-5.3-Flash); every number was produced by
runs on my machine and re-checked from the raw distribution files.
[publication.zip](https://github.com/user-attachments/files/34554048/publication.zip)関連リンク
インストール・モデル・リリースへのサイト内リンク。