Issues / #2670
#2670 Strata Benchmark Report — Swift 1.5 Flash Next (GSQ-RCO IQ3_S) - multiple parameter combinations
closed · @DrLove73 · 0 comments · View on GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsSecurity
Description
# 📊 Strata Benchmark Report — Swift 1.5 Flash Next (GSQ-RCO IQ3_S) **Date:** October 15, 2026 **Author:** @llm.plnet.rs **Hardware:** NVIDIA RTX PRO 4000 Blackwell (24 GB) · Intel Xeon Gold 6138 (60 cores) · 220 GB RAM · PCIe 3.0 x16 **Software:** Strata v0.1.44.1 (commit `296b59d2`) · Engine 0.1.44.1 **GPU Driver:** NVIDIA 580.105.08 · **CUDA:** 13.0 **OS:** RHEL 10.2 (kernel 6.12.0-211.49.1.el10_2.x86_64) --- ## 📋 Table of Contents 1. [Model & Environment](#-model--environment) 2. [Methodology](#-methodology) 3. [Run ID Legend](#-run-id-legend) 4. [Executive Summary](#-executive-summary) 5. [Speed Results (Raw Data)](#-speed-results-raw-data) 6. [NIAH Results (512K)](#-niah-results-512k) 7. [Parallel Tests](#-parallel-tests) 8. [Recommendations](#-recommendations) 9. [Reproduction](#-reproduction) 10. [Acknowledgments](#-acknowledgments) --- ## 🧠 Model & Environment | Property | Value | |---|---| | **Model** | Swift 1.5 Flash Next (UkisAI) | | **Quantization** | GSQ-RCO IQ3_S | | **Quality (KLD)** | 0.145 | | **Native context** | 262,144 tokens | | **Extended context** | 1,048,576 tokens (YaRN, auto-applied) | | **Experts in RAM** | 46.84 GiB | | **Expert cache (VRAM)** | 13.8–16.0 GiB (varies by context) | | **Download size** | ~84 GB (2 shards) | ### Host | Component | Specification | |---|---| | GPU | NVIDIA RTX PRO 4000 Blackwell (24 GiB VRAM, CC 12.0) | | GPU driver | NVIDIA 580.105.08 | | CUDA | 13.0 | | CPU | Intel Xeon Gold 6138 @ 2.00 GHz (60 cores / 60 threads, AVX2) | | RAM | 220 GB (upgrade to 400 GB planned) | | Storage | NVMe (model cache) | | PCIe | 3.0 x16 | | OS | RHEL 10.2 (kernel 6.12.0-211.49.1.el10_2.x86_64) | --- ## 🔬 Methodology Three independent test suites were used: | Test | Script | What it measures | |---|---|---| | **Speed** | `tools/speed_probe.py` | TTFT and decode tok/s on prompts at 96% of `--max-context` (Paul Graham Essays corpus) | | **Quality** | `tools/niah_test.py` | Needle-in-a-Haystack retrieval at 10%, 50%, 90% depth (via `needlehaystack`) | | **Parallel** | `tools/test_parallel.py` | Aggregate throughput, TTFT and decode for N simultaneous requests staggered 2–5 s apart | **Fixed parameters across all runs:** `--kv-resident 32768` · `--vision on` · `--spec 4` · `--adapt-decay 0.97` · `--adapt-swaps 160` · `--pool-workers 15` · `--expert-cache auto` · `--mtp rt` **Sampling:** Temperature 0, greedy decoding. NIAH uses `reasoning_budget_tokens=100` and `max_tokens=2048`. --- ## 🏷 Run ID Legend Every run is identified by a 9-field code. Example: **`4-1-y-i-32768-32-4-97-160`** | # | Field | Values | Meaning | |---|---|---|---| | 1 | **Context** | `1`=128K · `2`=256K · `3`=384K · `4`=512K · `5`=1M | Max context window | | 2 | **Batch** | `1` `2` `4` | Parallel slots (`--batch N`) | | 3 | **Vision** | `y`=on · `n`=off | Vision encoder enabled | | 4 | **KV type** | `i`=int8 · `k`=k8v4 · `q`=q4_0 · `f`=fp16 | KV cache quantization | | 5 | **Prefill** | `a`=auto · `a32`=auto:32768 · `a64`=auto:65536 · `32`=32768 · `64`=65536 | Prefill chunk size | | 6 | **KV resident** | `32`=32768 · `64`=65536 · `131`=131072 | Resident KV window | | 7 | **Spec** | `4` | Speculative tokens per window | | 8 | **Adapt decay** | `97`=0.97 · `95`=0.95 | Adaptive tier decay | | 9 | **Adapt swaps** | `160` · `200` | Adaptive swaps per step | **Read as:** `4-1-y-i-32768-32-4-97-160` = 512K context, batch 1, vision on, KV int8, prefill 32768, KV-resident 32768, spec 4, decay 0.97, swaps 160. --- ## 🎯 Executive Summary ### Best configuration per context | Context | Best config | TTFT | Decode | Quality | |---|---|---|---|---| | **128K** | `1-1-y-k-32768-32-4-97-160` | **58.8 s** | **65.5 t/s** | — | | **256K** | `2-1-y-k-32768-32-4-97-160` | **123.4 s** | **71.3 t/s** | — | | **512K** | `4-1-y-i-32768-32-4-97-160` | **265.3 s** | **66.3 t/s** | ✅ 3/3 | | **1M** | `5-1-y-q-a64-32-4-97-160` | **662.5 s** | **52.1 t/s** | — | ### Key findings - 🥇 **`--prefill 32768`** cuts TTFT by 25–30% vs `auto` on 256K–512K, with no cost to decode. - 🥈 **`--kv-resident 32768`** beats `65536` on 256K–512K decode; the opposite is true on 1M. - 🥉 **`--kv k8v4`** is fastest on 128K–256K; **`int8`** on 512K; **`q4_0`** on 1M. - ⚡ **`--batch 4`** gives +35% aggregate throughput on ≤256K, neutral on 512K, **counterproductive on 1M**. - ✅ **Quality (NIAH) is unaffected** by KV type, batch size, or context length on 512K — all runs scored 3/3. --- ## 🚀 Speed Results (Raw Data) **Legend:** - **ctx** — max context window (tokens) - **TTFT** — time to first token (seconds, lower is better) - **decode** — generation speed (tok/s, higher is better) ```text run_id ctx TTFT decode ───────────────────────────────────────────────────────── 2-1-y-k-a-32-4-97-160 262144 155.0 72.94 ★ highest 256K decode 2-1-y-k-32768-32-4-97-160 262144 123.4 71.26 ★ best 256K balance 2-1-y-i-32768-32-4-97-160 262144 117.4 68.24 2-1-y-q-a-32-4-97-160 262144 148.9 67.84 4-1-y-i-32768-32-4-97-160 524288 265.3 66.34 ★ best 512K balance 2-1-y-i-32768-64-4-97-160 262144 112.0 65.64 1-1-y-k-32768-32-4-97-160 131072 58.8 65.54 ★ best 128K balance 2-1-y-i-a-32-4-97-160 262144 156.6 64.85 4-1-y-i-a-64-4-97-160 524288 345.2 64.48 1-1-y-i-a-64-4-97-160 131072 67.6 64.41 4-1-y-q-32768-64-4-97-160 524288 264.0 64.29 1-1-y-k-a-64-4-97-160 131072 71.2 64.07 2-1-y-q-a-64-4-97-160 262144 145.3 63.91 4-1-y-k-32768-32-4-97-160 524288 262.8 63.77 2-1-y-k-32768-64-4-97-160 262144 117.9 63.53 4-1-y-k-a-64-4-97-160 524288 351.1 63.47 1-1-y-q-32768-32-4-97-160 131072 58.6 63.46 4-1-y-q-32768-32-4-97-160 524288 272.1 62.32 2-1-y-i-a-64-4-97-160 262144 156.7 61.87 2-1-y-q-32768-64-4-97-160 262144 125.5 61.62 1-1-y-i-32768-64-4-97-160 131072 54.3 61.09 1-1-y-q-32768-64-4-97-160 131072 59.8 60.24 2-2-y-q-32768-32-4-97-160 262144 128.9 59.81 1-2-y-k-a-64-4-97-160 131072 75.3 59.66 1-1-y-i-32768-32-4-97-160 131072 55.4 59.52 4-1-y-i-a-32-4-97-160 524288 346.2 59.30 2-2-y-k-a-32-4-97-160 262144 159.5 59.18 1-1-y-q-a-64-4-97-160 131072 69.4 58.47 2-2-y-k-32768-32-4-97-160 262144 131.4 57.96 1-2-y-k-32768-32-4-97-160 131072 61.4 57.78 4-2-y-q-32768-32-4-97-160 524288 294.0 55.69 2-2-y-q-a-64-4-97-160 262144 155.9 55.33 1-2-y-i-a-32-4-97-160 131072 70.7 55.04 2-2-y-i-32768-64-4-97-160 262144 128.3 54.92 4-1-y-q-a-32-4-97-160 524288 345.1 54.60 1-2-y-q-32768-64-4-97-160 131072 64.3 54.56 1-2-y-i-32768-64-4-97-160 131072 59.9 53.96 1-2-y-q-32768-32-4-97-160 131072 60.2 53.95 1-2-y-k-a-32-4-97-160 131072 73.6 53.84 2-2-y-i-a-64-4-97-160 262144 151.2 53.16 1-2-y-q-a-64-4-97-160 131072 72.5 52.84 1-2-y-k-32768-64-4-97-160 131072 63.0 52.35 5-1-y-q-a64-32-4-97-160 1048576 662.5 52.06 ★ best 1M 1-1-y-k-a-32-4-97-160 131072 72.1 51.92 1-2-y-i-32768-32-4-97-160 131072 57.4 51.71 1-1-y-k-32768-64-4-97-160 131072 61.1 51.46 1-2-y-i-a-64-4-97-160 131072 71.8 50.84 1-2-y-q-a-32-4-97-160 131072 72.9 50.46 2-1-y-k-a-64-4-97-160 262144 160.5 48.94 5-1-y-i-a32-32-4-97-160 1048576 638.7 48.67 5-1-y-i-a32-64-4-97-160 1048576 657.5 48.63 5-1-y-q-a32-64-4-97-160 1048576 662.0 48.44 4-2-y-i-a-64-4-97-160 524288 343.5 47.23 4-2-y-k-a-32-4-97-160 524288 348.8 46.61 4-2-y-k-a-64-4-97-160 524288 358.5 46.18 4-1-y-k-a-32-4-97-160 524288 328.7 45.72 4-2-y-i-32768-64-4-97-160 524288 294.3 44.36 2-1-y-q-32768-32-4-97-160 262144 116.8 43.86 4-2-y-i-32768-32-4-97-160 524288 287.7 42.78 2-2-y-i-32768-32-4-97-160 262144 123.4 41.96 4-1-y-i-32768-64-4-97-160 524288 278.1 41.51 2-2-y-q-a-32-4-97-160 262144 151.5 41.14 4-2-y-i-a-32-4-97-160 524288 344.1 40.57 4-1-y-q-a-64-4-97-160 524288 336.2 40.42 2-2-y-i-a-32-4-97-160 262144 152.7 40.36 1-1-y-q-a-32-4-97-160 131072 70.5 40.20 4-1-y-k-32768-64-4-97-160 524288 274.3 39.88 2-2-y-k-32768-64-4-97-160 262144 132.6 37.13 2-2-y-k-a-64-4-97-160 262144 159.2 36.29 2-2-y-q-32768-64-4-97-160 262144 128.2 36.06 1-1-y-i-a-32-4-97-160 131072 66.6 32.71 5-1-y-q-a64-64-4-97-160 1048576 639.7 31.83 4-2-y-k-32768-64-4-97-160 524288 314.0 30.98 4-2-y-q-32768-64-4-97-160 524288 298.4 30.08 4-2-y-k-32768-32-4-97-160 524288 306.4 30.03 5-1-y-i-a64-32-4-97-160 1048576 628.3 29.63 5-1-y-q-a32-32-4-97-160 1048576 640.3 29.28 5-1-y-i-a64-64-4-97-160 1048576 643.0 28.95 4-2-y-q-a-64-4-97-160 524288 343.9 28.30 4-2-y-q-a-32-4-97-160 524288 345.6 28.25 ``` ### Observations by context | Context | Configurations | Best decode | Worst decode | Spread | |---|---|---|---|---| | 128K | 24 | 65.5 t/s | 32.7 t/s | 2.0× | | 256K | 24 | 72.9 t/s | 36.1 t/s | 2.0× | | 512K | 24 | 66.3 t/s | 28.3 t/s | 2.3× | | 1M | 8 | 52.1 t/s | 28.9 t/s | 1.8× | **Parallel impact:** Every config with batch ≥2 shows a 30–60% decode penalty on 512K, and a **50–80% penalty on 1M** (see Parallel Tests). --- ## 🎯 NIAH Results (512K) Needle-in-a-Haystack test with UUID needle at 10%, 50%, 90% depth. **Y** = found, **N** = not found. | Run ID | KV | Batch | d10 | d50 | d90 | Score | |---|---|---|---|---|---|---| | `4-1-y-i-32768-32-4-97-160` | int8 | 1 | ✅ | ✅ | ✅ | **3/3** | | `4-1-y-k-32768-32-4-97-160` | k8v4 | 1 | ✅ | ✅ | ✅ | **3/3** | | `4-1-y-q-32768-32-4-97-160` | q4_0 | 1 | ✅ | ✅ | ✅ | **3/3** | | `4-2-y-i-32768-32-4-97-160` | int8 | 2 | ✅ | ✅ | ✅ | **3/3** | | `4-2-y-k-32768-32-4-97-160` | k8v4 | 2 | ✅ | ✅ | ✅ | **3/3** | | `4-2-y-q-32768-32-4-97-160` | q4_0 | 2 | ✅ | ✅ | ✅ | **3/3** | ### Conclusion - ✅ **KV quantization does not affect recall.** Even aggressive `q4_0` retains the needle at 50% depth — no "lost in the middle" failure. - ✅ **Batch size does not affect recall.** Batch 2 scored identically to batch 1. - ✅ **512K is genuinely functional** — not just accepted by the engine, but actively used by the model. --- ## ⚡ Parallel Tests ### Test 1 — Short prompts (~32K tokens) **Setup:** 4 requests, 2 s stagger, `max_tokens=50` | Config | Wall | Throughput | Avg TTFT | Avg decode | |---|---|---|---|---| | 256K batch 1 | 58.3 s | 3.4 t/s | — | — | | **256K batch 4** | **43.2 s** | **4.6 t/s** | — | — | | 512K batch 1 | 58.9 s | 3.4 t/s | — | — | | **512K batch 4** | **43.7 s** | **4.6 t/s** | — | — | **Result:** Batch 4 gives **+35% throughput** and **−26% wall time** at the cost of **+19% per-request latency**. ✅ Worth it. ### Test 2 — Long prompts (~64K tokens) on 1M context **Setup:** 4 requests, 5 s stagger, `max_tokens=500`, `reasoning_budget_tokens=512` | Config | Wall | Throughput | Avg TTFT | Avg decode | Expert cache | |---|---|---|---|---|---| | **1M batch 1** | **53.1 s** | **6.04 t/s** | **33.9 s** | **60.2 t/s** | **96–98%** | | 1M batch 2 | 63.5 s | 5.34 t/s | 38.9 s | 14.8 t/s | ~85% | | 1M batch 4 | 108.6 s | 3.18 t/s | 69.7 s | 10.0 t/s | 79–85% | **Result:** Batch 2 and 4 are **catastrophic on 1M** — decode collapses from 60 t/s to 10–15 t/s. ### Why parallel fails at 1M | Metric | Batch 1 | Batch 4 | |---|---|---| | Experts in VRAM | 7,260 | 2,682 | | VRAM for expert cache | 13.77 GiB | 5.14 GiB | | PCIe fallback | +2.7% | **+15.6%** | **Root cause:** Each slot reserves its own 1M KV cache. With 4 sl
Related on this site
Quick links to related pages.