#3533 [Benchmark] L40S-48Q vGPU, Strata 0.2.0-beta: six Unsloth Qwen3.8-Flash-Next quants (UD-Q2_K_XL to Q8_0)

Descrição

## Problem and result

Which Unsloth quantization should you run on Strata 0.2.0 with one 48 GB card and every expert in RAM? This PR adds a results folder: `bench/results/2026-10-20-community-l40s-48q-0.2.0-unsloth-quants/`.

The machine is an NVIDIA L40S-48Q vGPU with a Xeon 8580 (32 vCPU) and 251 GiB RAM.
- Six GGUFs from one pinned revision, UD-Q2_K_XL to Q8_0, all on the same 0.2.0-beta engine with the same config.
- Q8_0 on 0.1.45 with the same config, as a reference.

**The 0.2.0 results are complete.** Tonight (2026-10-20 to 21) we are running the same battery on **Strata 0.2.1**: all six quantizations, interleaved 0.2.0/0.2.1 starts on UD-Q4_K_XL, and a new 26-task 3D suite in which the model writes `bpy` scripts that Blender runs headless and a checker verifies. We will add those results to this PR in about 14 hours.

| Model | Decode tok/s 8K / 32K / 128K | Prompt tok/s 32K / 128K | KL against Q8_0 (60 / 160 prompts) | HumanEval+ plus | needles 128K |
|---|---|---|---|---|---|
| UD-Q2_K_XL | 126 / 122 / 111 | 3,194 / 3,425 | 0.202 / 0.405 | 89.0 | 3/3 |
| UD-Q3_K_XL | 107 / 108 / 105 | 3,484 / 3,417 | 0.108 / 0.158 | 87.2 | 3/3 |
| UD-Q4_K_XL | 98 / 99 / 96 | 2,828 / 2,739 | 0.037 / 0.065 | 91.5 | 3/3 |
| UD-Q5_K_XL | 89 / 90 / 83 | 2,288 / 2,167 | 0.023 / 0.034 | 89.6 | 3/3 |
| UD-Q6_K_XL | 82 / 82 / 78 | 2,037 / 1,928 | 0.015 / 0.037 | 89.0 | 3/3 |
| Q8_0 | 75 / 73 / 72 | 1,703 / 1,613 | - | 89.0 | 3/3 |
| Q8_0, 0.1.45 | 74 / 70 / 70 | 1,712 / 1,616 | 0.012 / 0.024 (against 0.2.0) | 90.2 | - |
| UD-Q4_K_XL, tail skip 7 | 97 / 103 / 96 | 2,778 / 2,726 | - | 90.2 | - |

Findings:
- **UD-Q4_K_XL is the knee:** low KL, and about 30% faster than Q8_0.
- **UD-Q2/Q3_K_XL drift 3 to 6 times further** from Q8_0, while scoring about the same on the task suites. GSM8K, IFEval, MATH geometry and BBH all reach 95% or more on every quantization, so they do not separate quantizations here.
- **0.2.0 is not slower than 0.1.45** with the same config: decode is equal or 2-5% faster, prompt speed is equal.
- **`STRATA_ROUTE_TAIL_SKIP` makes no speed difference on this card.** The setup's `=0` and the default 7 give the same decode on UD-Q4_K_XL, because 96-98% of expert lookups hit the GPU cache. Our earlier guess, that it explained why decode is about 20% below our older UD-Q4_K_XL numbers, was wrong.
- **lm-eval's default 256 tokens truncates GSM8K** answers of a thinking model before the final number. UD-Q2_K_XL scores 74% at 256 tokens and 98% at 4,096.
- **BBH geometric_shapes is capped at 80%:** the same 50 documents labelled "(K) trapezoid" fail on every quantization.

## Related discussion

- #3269: `iq_pack.py` in the 0.2.0 control package could not find `gguf-py`. We worked around it with `STRATA_GGUF_PY`. Fixed in 0.2.1.
- No scope agreement needed: this is a results-only PR.

## Approach

- **Packages:** the 0.2.0-beta Linux control and cuda12 engine packages, checked against `SHA256SUMS.txt`, launched with `serve/server.py`.
- **Order:** one model and one request at a time, a fresh server per model.
- **Config:** our production run config with only the paths changed, plus the `env` block that `setup.py unsloth_default_env` writes for Unsloth files.
- **Speed:** single fresh requests at 8K, 32K and 128K, and `tools/strata_bench.py --runs 3`.
- **Quality:** full first-token distributions with `STRATA_DUMP_FIRST_LOGITS`, KL against Q8_0 on the same engine; `tools/needle_bench.py` at 128K.
- **Tasks:** lm-evaluation-harness 0.4.13 and EvalPlus 0.3.1 over the API, with thinking at the server default.
- The README gives exact settings, caps and sample sizes. The folder has the raw JSON, gzipped samples and logs, the configs (key removed) and the scripts.

## Validation

**Passed:**
- All nine runs loaded and served every request: six models, the 1M start check, the 0.1.45 reference and the tail-skip A/B. Event log: `engine-logs/pipeline-eventos.log.txt`; start-up lines: `results/*/info.json`.
- Every number in the tables can be regenerated from `results/` with `scripts/gerar-relatorio.py`, which writes `tables.md` and `summary.json`.
- GGUF sizes and SHA-256 at revision `766911a6b7369840a91dbcd95f9f997acaab6cd6` are in `model-manifest.json`.

**Failed:**
- None observed: no engine errors and no failed requests.
- One start-up attempt was refused by our own script because the GPU was still busy. It was rerun.

**Not run (yet):**
- Strata 0.2.1: running tonight, see above.
- Repeats on the same engine for a KL noise floor.
- Concurrency, thermals and clocks.
- A 3D/Blender task suite on 0.2.0 (it runs on 0.2.1 only).

## Limits

- **One vGPU machine; single requests per length per session.**
- **The first-token KL covers the prompt path only.** It does not cover the decode or verify path where the route tail skip acts.
- **The two Q8_0 runs (0.1.45 and 0.2.0) differ by KL 0.012 / 0.024.** That is the size of the UD-Q6 → Q8 difference, so read the KL of UD-Q5/Q6 with that in mind.
- **A host problem after our RAM upgrade, now fixed.** After the VM went from 251 to 503 GiB, large pinned registrations crashed the vGPU guest driver (a kernel Oops in `rm_ioctl` during `cudaHostRegister`). A bare test reproduced it without Strata.
  - **Cause:** the VM's new memory was on the host's other socket, away from the GPU (NUMA).
  - **Fix:** the hosting team moved the memory. After that, 1-192 GiB pinned and Q8_0 at 1M both work.
  - The speed and quality numbers were taken at 251 GiB. Only the reference's GSM8K/IFEval/MATH/BBH and the tail-skip A/B ran at 503 GiB.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01E3U3Q9QVVJFxinHHkEaRu6