#5610 [Benchmark] Independent distributional quality: Strata vs llama.cpp vs BF16 truth, 399,600 teacher-forced positions

closed · @Andrey0710 · 1 comments · View on GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationWindows

Description

Result (short)

I ran an independent, teacher-forced distributional comparison of Strata against llama.cpp
on the same GGUF files, with a full-precision (BF16) run of the same model as the ground
truth, on a ~400,000-position public-domain corpus. Bottom line:

**No distributional degradation was found.** On every quant where Strata was measured,
its agreement with the BF16 truth is equal to or better than the reference llama.cpp
build's on the same file:

| Quant | llama.cpp: top-1 / KL | Strata: top-1 / KL | Verdict |
| --- | --- | --- | --- |
| UD-Q4_K_XL | 94.02% / 0.0526 | **94.39% / 0.0461** | Strata closer to truth |
| UD-IQ4_XS | 92.29% / 0.0852 | **92.53% / 0.0808** | Strata closer to truth |
| IQ3_S | 89.80% / 0.1577 | 89.81% / 0.1580 | parity |

- **Quantization, not the engine, dominates the divergence.** The quant ladder
  (llama.cpp vs BF16: KL 0.053 to 0.233 across Q4_K_XL to IQ3_XXS) is 2-4x larger than the
  between-engine disagreement on the same file (KL 0.015-0.028, top-1 95.5-96.6%).
- Strata's engine is bit-deterministic in the fixed yardstick configuration (two identical
  runs: KL 0.0000), so every measured divergence is real, not noise.

**Comparable with the project's published quality tables.** The measurements use the
project's own yardstick (`tools/quant_quality.py`, teacher-forced top-64), so the numbers
are directly comparable with the quality data in docs/UNSLOTH_Q4.md and docs/QUANTS.md -
with a new, larger corpus and a BF16 (not Q8_0) reference.

## Setup

- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation 96 GB; RAM 253 GB; Windows, CUDA 13.3.
- Strata 0.2.12, engine 0.1.58 (release binary).
- llama.cpp: unsloth b12128-mix-2f23ac9 CUDA build ("the reference").
- Weights: unsloth UD-IQ4_XS (87.2 GB, 4 shards, **Dynamic V3.1**, shards 2026-11-04),
  unsloth UD-Q4_K_XL (111.3 GB, **pre-V3.1** revision of 2026-08-29, downloaded 2026-10-21 - the repo's Q4_K_XL was re-packed on 2026-11-04 into the mix-n-match engram layout, so our copy is the older revision), ISTA-DASLab
  GSQ-RCO IQ3_S (83.6 GB) and IQ3_XXS (75.8 GB), unsloth BF16 (345.7 GB, 8 shards).
  GGUF metadata does not carry the unsloth quant generation - shard dates and the HF
  release timeline are the provenance (files are updated in place).
- Corpus "standard-v2": 400 segments x 1,000 tokens, interleaved public sources -
  llama.cpp/ggml sources (MIT), llama.cpp/Strata docs (MIT), Pride and Prejudice and
  War and Peace (public domain). SHA256 of every source in the manifest.

## Method

Teacher-forced comparison on identical token id sequences. Per position each side writes its
top-64 log-probabilities (Strata: `STRATA_LOGPOS` / `STRATA_LOGPOS_TOPK=64` through the
project's own yardstick `tools/quant_quality.py run` with its fixed configuration - expert-cache auto, FP16 KV, no adaptation, no draft/speculation; llama.cpp: a small custom
dumper over `llama_decode`/`llama_get_logits_ith` writing the same `STRATA_LOGPOS` row
format). Metrics: top-1 argmax agreement, KL(REF||RUN) over REF's top-64 renormalized, KL p99,
and the text's perplexity under each side.

The BF16 ground truth ran through llama.cpp with the MoE experts and the per-layer
n-gram table streamed (mmap + lazy reads + tensor buffer-type overrides), ~33 s per 1,000
positions. Strata's engine was bit-identical across two identical runs (KL 0.0000), which
sets the noise floor at zero.

## What is attached

- Full report with all tables, the anatomy of divergences (near-tie share,
  gap distributions, worst positions), the depth analysis, and the limitations: `REPORT-FULL.md`
- Raw distributions (TSV + sidecars) and all comparison JSONs - on request (~4 GB)
- Scripts: corpus builder, the llama.cpp reference dumper (C++), alignment and analysis
  scripts, exact run commands
- Token ids of the public corpus + manifest with SHA256 of every source file

## Scope

Distributional comparison only (teacher-forced positions, fixed yardstick configuration,
single GPU, Windows). The full list of limitations is in the attached report.

## Notes

- The measured engine-vs-engine disagreement (KL 0.015-0.028) is 2-4x smaller than the
  quantization cost itself. On the published corpus, 3.92% of positions diverge in argmax;
  2.49% carry a gap >= 0.2 nat and 0.45% a gap >= 1.0 nat - so the earlier near-ties-only
  reading of docs/UNSLOTH_Q4.md softens at scale, but the fidelity-to-truth outcome does
  not change.
- Happy to share the full raw distributions, re-run anything on request, or adjust the
  methodology if the maintainers see a flaw.
- Preparation note: the benchmark harness, the reference dumper and the analysis were
  built and verified with an AI assistant (GLM-5.3-Flash); every number was produced by
  runs on my machine and re-checked from the raw distribution files.

[publication.zip](https://github.com/user-attachments/files/34554048/publication.zip)

Related on this site

Quick links to related pages.