#2446 Community benchmark: 2x NVIDIA A2 16 GB (sm_86, PCIe only), Xeon Gold 6444Y, 62 GiB RAM - UD-IQ4_XS and Q8_0, 64K to 1M, v0.1.42

Descrição

# Community benchmark on 2x NVIDIA A2 16 GB (sm_86, PCIe only), 62 GiB RAM

**Measured 2026-10-10 to 2026-10-14 on Strata v0.1.42. UD-IQ4_XS and Q8_0 of Qwen3.8-Flash-Next, contexts 64K to 1M, speed and answer quality. More quantizations are still running.**

## Context

- This is a lab VM that sat idle for months. We are using it for a long, repeatable battery on Strata: speed and long-context behaviour, and (our main focus) the **quality of the answers of each quantization against Q8_0**.
- We also run Strata on a separate production VM. The lab VM is deliberately isolated from it, so we can run for days without disturbing anything.
- We want to help the project: give back measurements in the format you ask for, and report the rough edges we met. Nothing here is urgent. We can rerun any of it on a newer build.
- The battery is automated and this text was drafted by an AI assistant (Claude Code) working from our logs. The account owner reads and answers replies on this thread.

## Related threads from the same battery

- Bug (new): STRATA_DUMP_FIRST_LOGITS writes nothing with `--pipeline-windows 2`, silently: #2445
- Data point added to the checkpoint-save error thread (#1932): https://github.com/Niko1221/Strata/issues/1932#issuecomment-6170938660
- Data point added to the CUDA OOM on a 2-GPU split thread (#2408): https://github.com/Niko1221/Strata/issues/2408#issuecomment-6170939159
- Related existing threads we did not touch: #2065 (VRAM headroom), #2275 (multi-user direction; we may later explore one GPU per request with a shared RAM copy of the model, and would report that separately).

## Hardware and software

- GPU: 2x NVIDIA A2, 15,356 MiB each, 60 W, Ampere sm_86. PCIe gen4, link width **x8** each (max x16), no NVLink, no P2P. Layer split only.
- CPU/RAM/storage: Xeon Gold 6444Y, 16 vCPUs (VMware guest, 1 socket, AVX-512, AMX), 62 GiB RAM, 500 GB virtual disk (fio: 5,758 MiB/s sequential read, 4K random read 462 MiB/s at queue depth 32).
- OS: Ubuntu 24.04.4, kernel 6.8.0-139, driver 570.211.01, CUDA 12.8.
- Strata: tag v0.1.42, commit `61b3fb5dd3f1e8ec09cf7e4e05208bc6d3c46406`, **source build for sm_86** (setup found no ready-made engine for this architecture and compiled one). Test engine built with `-DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_EXPERIMENTAL_SM60=ON`; the 4 exactness tests for the Q6_K/Q8_0 pieces passed on Ampere.
- Background load: none (nothing else runs on the VM). One request at a time (no concurrency tests yet).

## Model and configuration

- Model: `unsloth/Qwen3.8-Flash-Next-GGUF` (revision main, Oct 2026): **UD-IQ4_XS** (93.7 GB) and **Q8_0** (188.2 GB, packed with `iq_pack.py --compat-bf16`; the experts are read straight from the GGUF, no `experts.bin`); vision enabled (mmproj BF16).
- Same settings for every test: 2 GPUs, `--layer-split auto`, `--pipeline-windows 2`, `--prefill auto`, `--spec 4 --spec-min-p 0.5` with the MTP block, `--kv int8 --kv-resident 32768`, `--resident-budget-gib 35`, `--expert-cache auto`, `--max-context` 65536 / 131072 / 262144 / 524288 / 1048576 (YaRN above 262144).
- `--vram-reserve-mib`: **700 for UD-IQ4_XS, 1000 for Q8_0** (see the OOM section; the two models are therefore not at identical reserve).
- Reasoning: off. Direct tests send `chat_template_kwargs: {"enable_thinking": false}`; the evals use `reasoning_effort: none` in the config. Greedy (temperature 0). Cold disk cache at every server start.
- Server start: 60 to 75 s.

## Results

Speed numbers are the server's own timings. "Full context" = a prompt filling 97% of the window, then a 512-token continuation that reuses it. One run per point for these; three runs for the short-prompt sweep. `hit_rate` as reported (VRAM share, excludes `--pcie-frac` experts, see #588).

| Context | Model | Prompt tokens | Prompt tok/s | Time to first token | Decode tok/s (512 tokens, full context) | hit_rate |
|---|---|---:|---:|---:|---:|---:|
| 64K | UD-IQ4_XS | 64,489 | 742 | 87 s | 19.3 | 0.91 |
| 64K | Q8_0 | 64,488 | 192 | 336 s | 6.2 | 0.70 |
| 128K | UD-IQ4_XS | 124,091 | 767 | 162 s | 18.6 | 0.89 |
| 128K | Q8_0 | 124,091 | 183 | 680 s | 5.9 | 0.69 |
| 256K | UD-IQ4_XS | 250,616 | 742 | 338 s | 16.9 | 0.92 |
| 256K | Q8_0 | 250,616 | 189 | 1,329 s | 4.9 | 0.71 |
| 512K | UD-IQ4_XS | 497,579 | 591 | 843 s | 15.1 | 0.90 |
| 512K | Q8_0 | 497,578 | 178 | 2,800 s | 4.6 | 0.71 |
| 1M | UD-IQ4_XS | 1,041,873 | 398 | 2,620 s | 12.0 | 0.88 |
| 1M | Q8_0 | 1,041,873 | 166 | 6,274 s | 3.9 | 0.67 |

Short prompts, 64K config, median of 3: UD-IQ4_XS 4K prompt 352 tok/s prompt / 25.0 decode, 32K prompt 651 / 23.7; Q8_0 4K 76 / 6.7, 32K 185 / 6.7. Q8_0 (188 GB) does not fit in 62 GiB, so it reads experts from the SSD (about 36 to 42 GB read per request in the 4K and 32K sweeps, 256 tokens generated).

**Needle recall:** three hidden codes at 10/50/90% depth, recovered in order: 3/3 at every context from 64K to 1M on both models. `tools/needle_bench.py` on Q8_0: 15/15 found (32K, 128K, 262K, 400K, 500K x three depths; 200 s to 2,678 s each). Not yet run on UD-IQ4_XS.

**Parallelism without NVLink** (UD-IQ4_XS, 64K config, median of 3 decode tok/s at 4K / 16K prompts): default layer split 22.1 / 23.6; `--pipeline-windows 2` 23.4 / 26.0; peer without P2P 20.2 / 25.0. We adopted `--pipeline-windows 2` (+6 to +10% decode).

**1 GPU vs 2 GPUs** (UD-IQ4_XS, 256K config): sweep decode 21.5 / 21.1 / 20.0 (1 GPU) vs 22.5 / 22.6 / 21.5 (2 GPUs) at 4K / 32K / 128K; full-context prompt 411 vs 742 tok/s, decode 15.8 vs 16.9. `hit_rate` 0.62 to 0.66 (1 GPU) vs 0.89 to 0.92 (2 GPUs). The second card mostly helps prompt reading. Q8_0 on one GPU: 128K sweep 179 tok/s prompt, 5.6 to 5.8 decode, `hit_rate` 0.45.

### Answer quality (end-to-end correctness checks, one run, temperature 0)

| Check | Q8_0 | UD-IQ4_XS |
|---|---:|---:|
| HumanEval base / plus (164) | 95.1 / 92.7 | 94.5 / 92.7 |
| MBPP base / plus (378) | 93.4 / 78.8 | 92.6 / 79.4 |
| GSM8K flexible / strict (250) | 94.0 / 92.8 | 94.4 / 94.0 |
| IFEval prompt strict / loose (250) | 80.8 / 84.4 | 81.2 / 86.0 |
| 24 images with known answers (text 9 to 24 px, counting, table, colour/position) | 18/24 | 18/24 |

Paired exact McNemar tests on the same problems: p >= 0.42 on every metric, so no difference is detectable at these sample sizes. This is consistent with, but weaker than, the teacher-forced KL numbers in the 0.1.44 release notes (UD-IQ4_XS 88.9% same top token against Q8_0): task accuracy hides a measurable divergence. We are collecting first-token KL on 220 prompts too and will add it.

## What needed a workaround

1. **Engine exits with CUDA out of memory in small allocations, Q8_0, 2x16 GB, `--vram-reserve-mib 700`.** 128K: `mtp prefill: the input records do not fit` (cudaMalloc, `src/core/mtp.cpp` ~1138/1180 in v0.1.42), and once `mtp: step capture: no error` (no cause). 256K: `saving a checkpoint part failed` to the client; the reason (`checkpoint save: conversation snapshot running-state copy: out of memory`) is only in the engine's stderr. "VRAM free at start" does not predict it: UD-IQ4_XS passes with 137 to 149 MiB free, Q8_0 at 64K passes with 29 MiB, Q8_0 at 128K fails with 383 MiB. 128K reproduced 3 of 3. At 128K it passes with `--vram-reserve-mib 1500`, with `--pipeline-windows 0`, or with `--prefill 1024` (one run each). We set `--vram-reserve-mib 1000` for every model: Q8_0 then ran 64K to 1M and over a thousand eval requests with no engine error (one run per context). See the two comments linked above.
2. **STRATA_DUMP_FIRST_LOGITS silently does nothing with `--pipeline-windows 2`:** #2445. Workaround: collect with `--pipeline-windows 0`.
3. **Notes for anyone scoring a slow model with common harnesses (not Strata bugs):**
   - `evalplus` (openai backend) uses `signal.alarm(100)` in `gen/util/openai_request.py` and retries the same request forever when it fires. At ~5 tok/s a reply over ~500 tokens never fits; the server logs `done: ... (cancel, cancel=True)` at ~101 s again and again (66 of 98 requests) and the run sat on one problem for ~2 h. We raised it to 7200 s.
   - `lm_eval` `local-chat-completions` defaults to a 300 s timeout; IFEval generates up to 1280 tokens (~256 s at 5 tok/s). We set `timeout=7200`.
   - Pass chat replies through `evalplus.sanitize` before scoring: code fences made HumanEval score 0.0% in an earlier session.

## Already covered upstream (we found this by reading the docs afterwards)

Thinking control through the API (`DETAILS.md`), the meaning of `hit_rate` (#588), Q6_K experts being built by default (`QUANTS.md`, 0.1.44), and what `--compat-bf16` rounds (`tools/iq_pack.py` docstring). We had listed these as questions; they are answered, so we are not opening issues for them.

## Not tested / limitations

- Only v0.1.42. Not re-run on 0.1.43 to 0.1.44.1, which may already change item 1.
- One request at a time. No concurrency, `--batch`, or conversation-cache tests yet.
- Single run for the long-context and needle points; UD-IQ4_XS ran with reserve 700 and Q8_0 with 1000.
- Still queued: UD-Q6_K_XL, UD-Q5_K_XL, UD-Q4_K_XL, UD-Q3_K_XL, UD-Q2_K_XL; and the first-token KL pass.

## Data

Per-request JSON, engine logs and the scripts are kept with the results and can be shared on request or as a pull request under `bench/results/` if you prefer that format. We can also rerun the same battery on a newer build and send the diff.