#2670 Strata Benchmark Report — Swift 1.5 Flash Next (GSQ-RCO IQ3_S) - multiple parameter combinations

closed · @DrLove73 · 0 comentarios · En GitHub
Benchmarks Setup & install Server & API NVIDIA / CUDA Models & quants Security
Descripción

# 📊 Strata Benchmark Report — Swift 1.5 Flash Next (GSQ-RCO IQ3_S)

**Date:** October 15, 2026
**Author:** @llm.plnet.rs
**Hardware:** NVIDIA RTX PRO 4000 Blackwell (24 GB) · Intel Xeon Gold 6138 (60 cores) · 220 GB RAM · PCIe 3.0 x16
**Software:** Strata v0.1.44.1 (commit `296b59d2`) · Engine 0.1.44.1
**GPU Driver:** NVIDIA 580.105.08 · **CUDA:** 13.0
**OS:** RHEL 10.2 (kernel 6.12.0-211.49.1.el10_2.x86_64)

---

## 📋 Table of Contents

1. [Model & Environment](#-model--environment)
2. [Methodology](#-methodology)
3. [Run ID Legend](#-run-id-legend)
4. [Executive Summary](#-executive-summary)
5. [Speed Results (Raw Data)](#-speed-results-raw-data)
6. [NIAH Results (512K)](#-niah-results-512k)
7. [Parallel Tests](#-parallel-tests)
8. [Recommendations](#-recommendations)
9. [Reproduction](#-reproduction)
10. [Acknowledgments](#-acknowledgments)

---

## 🧠 Model & Environment

| Property | Value |
|---|---|
| **Model** | Swift 1.5 Flash Next (UkisAI) |
| **Quantization** | GSQ-RCO IQ3_S |
| **Quality (KLD)** | 0.145 |
| **Native context** | 262,144 tokens |
| **Extended context** | 1,048,576 tokens (YaRN, auto-applied) |
| **Experts in RAM** | 46.84 GiB |
| **Expert cache (VRAM)** | 13.8–16.0 GiB (varies by context) |
| **Download size** | ~84 GB (2 shards) |

### Host

| Component | Specification |
|---|---|
| GPU | NVIDIA RTX PRO 4000 Blackwell (24 GiB VRAM, CC 12.0) |
| GPU driver | NVIDIA 580.105.08 |
| CUDA | 13.0 |
| CPU | Intel Xeon Gold 6138 @ 2.00 GHz (60 cores / 60 threads, AVX2) |
| RAM | 220 GB (upgrade to 400 GB planned) |
| Storage | NVMe (model cache) |
| PCIe | 3.0 x16 |
| OS | RHEL 10.2 (kernel 6.12.0-211.49.1.el10_2.x86_64) |

---

## 🔬 Methodology

Three independent test suites were used:

| Test | Script | What it measures |
|---|---|---|
| **Speed** | `tools/speed_probe.py` | TTFT and decode tok/s on prompts at 96% of `--max-context` (Paul Graham Essays corpus) |
| **Quality** | `tools/niah_test.py` | Needle-in-a-Haystack retrieval at 10%, 50%, 90% depth (via `needlehaystack`) |
| **Parallel** | `tools/test_parallel.py` | Aggregate throughput, TTFT and decode for N simultaneous requests staggered 2–5 s apart |

**Fixed parameters across all runs:**
`--kv-resident 32768` · `--vision on` · `--spec 4` · `--adapt-decay 0.97` · `--adapt-swaps 160` · `--pool-workers 15` · `--expert-cache auto` · `--mtp rt`

**Sampling:** Temperature 0, greedy decoding. NIAH uses `reasoning_budget_tokens=100` and `max_tokens=2048`.

---

## 🏷 Run ID Legend

Every run is identified by a 9-field code. Example: **`4-1-y-i-32768-32-4-97-160`**

| # | Field | Values | Meaning |
|---|---|---|---|
| 1 | **Context** | `1`=128K · `2`=256K · `3`=384K · `4`=512K · `5`=1M | Max context window |
| 2 | **Batch** | `1` `2` `4` | Parallel slots (`--batch N`) |
| 3 | **Vision** | `y`=on · `n`=off | Vision encoder enabled |
| 4 | **KV type** | `i`=int8 · `k`=k8v4 · `q`=q4_0 · `f`=fp16 | KV cache quantization |
| 5 | **Prefill** | `a`=auto · `a32`=auto:32768 · `a64`=auto:65536 · `32`=32768 · `64`=65536 | Prefill chunk size |
| 6 | **KV resident** | `32`=32768 · `64`=65536 · `131`=131072 | Resident KV window |
| 7 | **Spec** | `4` | Speculative tokens per window |
| 8 | **Adapt decay** | `97`=0.97 · `95`=0.95 | Adaptive tier decay |
| 9 | **Adapt swaps** | `160` · `200` | Adaptive swaps per step |

**Read as:** `4-1-y-i-32768-32-4-97-160` = 512K context, batch 1, vision on, KV int8, prefill 32768, KV-resident 32768, spec 4, decay 0.97, swaps 160.

---

## 🎯 Executive Summary

### Best configuration per context

| Context | Best config | TTFT | Decode | Quality |
|---|---|---|---|---|
| **128K** | `1-1-y-k-32768-32-4-97-160` | **58.8 s** | **65.5 t/s** | — |
| **256K** | `2-1-y-k-32768-32-4-97-160` | **123.4 s** | **71.3 t/s** | — |
| **512K** | `4-1-y-i-32768-32-4-97-160` | **265.3 s** | **66.3 t/s** | ✅ 3/3 |
| **1M** | `5-1-y-q-a64-32-4-97-160` | **662.5 s** | **52.1 t/s** | — |

### Key findings

- 🥇 **`--prefill 32768`** cuts TTFT by 25–30% vs `auto` on 256K–512K, with no cost to decode.
- 🥈 **`--kv-resident 32768`** beats `65536` on 256K–512K decode; the opposite is true on 1M.
- 🥉 **`--kv k8v4`** is fastest on 128K–256K; **`int8`** on 512K; **`q4_0`** on 1M.
- ⚡ **`--batch 4`** gives +35% aggregate throughput on ≤256K, neutral on 512K, **counterproductive on 1M**.
- ✅ **Quality (NIAH) is unaffected** by KV type, batch size, or context length on 512K — all runs scored 3/3.

---

## 🚀 Speed Results (Raw Data)

**Legend:**
- **ctx** — max context window (tokens)
- **TTFT** — time to first token (seconds, lower is better)
- **decode** — generation speed (tok/s, higher is better)

```text
run_id                       ctx        TTFT     decode
─────────────────────────────────────────────────────────
2-1-y-k-a-32-4-97-160        262144     155.0    72.94   ★ highest 256K decode
2-1-y-k-32768-32-4-97-160    262144     123.4    71.26   ★ best 256K balance
2-1-y-i-32768-32-4-97-160    262144     117.4    68.24
2-1-y-q-a-32-4-97-160        262144     148.9    67.84
4-1-y-i-32768-32-4-97-160    524288     265.3    66.34   ★ best 512K balance
2-1-y-i-32768-64-4-97-160    262144     112.0    65.64
1-1-y-k-32768-32-4-97-160    131072      58.8    65.54   ★ best 128K balance
2-1-y-i-a-32-4-97-160        262144     156.6    64.85
4-1-y-i-a-64-4-97-160        524288     345.2    64.48
1-1-y-i-a-64-4-97-160        131072      67.6    64.41
4-1-y-q-32768-64-4-97-160    524288     264.0    64.29
1-1-y-k-a-64-4-97-160        131072      71.2    64.07
2-1-y-q-a-64-4-97-160        262144     145.3    63.91
4-1-y-k-32768-32-4-97-160    524288     262.8    63.77
2-1-y-k-32768-64-4-97-160    262144     117.9    63.53
4-1-y-k-a-64-4-97-160        524288     351.1    63.47
1-1-y-q-32768-32-4-97-160    131072      58.6    63.46
4-1-y-q-32768-32-4-97-160    524288     272.1    62.32
2-1-y-i-a-64-4-97-160        262144     156.7    61.87
2-1-y-q-32768-64-4-97-160    262144     125.5    61.62
1-1-y-i-32768-64-4-97-160    131072      54.3    61.09
1-1-y-q-32768-64-4-97-160    131072      59.8    60.24
2-2-y-q-32768-32-4-97-160    262144     128.9    59.81
1-2-y-k-a-64-4-97-160        131072      75.3    59.66
1-1-y-i-32768-32-4-97-160    131072      55.4    59.52
4-1-y-i-a-32-4-97-160        524288     346.2    59.30
2-2-y-k-a-32-4-97-160        262144     159.5    59.18
1-1-y-q-a-64-4-97-160        131072      69.4    58.47
2-2-y-k-32768-32-4-97-160    262144     131.4    57.96
1-2-y-k-32768-32-4-97-160    131072      61.4    57.78
4-2-y-q-32768-32-4-97-160    524288     294.0    55.69
2-2-y-q-a-64-4-97-160        262144     155.9    55.33
1-2-y-i-a-32-4-97-160        131072      70.7    55.04
2-2-y-i-32768-64-4-97-160    262144     128.3    54.92
4-1-y-q-a-32-4-97-160        524288     345.1    54.60
1-2-y-q-32768-64-4-97-160    131072      64.3    54.56
1-2-y-i-32768-64-4-97-160    131072      59.9    53.96
1-2-y-q-32768-32-4-97-160    131072      60.2    53.95
1-2-y-k-a-32-4-97-160        131072      73.6    53.84
2-2-y-i-a-64-4-97-160        262144     151.2    53.16
1-2-y-q-a-64-4-97-160        131072      72.5    52.84
1-2-y-k-32768-64-4-97-160    131072      63.0    52.35
5-1-y-q-a64-32-4-97-160      1048576    662.5    52.06   ★ best 1M
1-1-y-k-a-32-4-97-160        131072      72.1    51.92
1-2-y-i-32768-32-4-97-160    131072      57.4    51.71
1-1-y-k-32768-64-4-97-160    131072      61.1    51.46
1-2-y-i-a-64-4-97-160        131072      71.8    50.84
1-2-y-q-a-32-4-97-160        131072      72.9    50.46
2-1-y-k-a-64-4-97-160        262144     160.5    48.94
5-1-y-i-a32-32-4-97-160      1048576    638.7    48.67
5-1-y-i-a32-64-4-97-160      1048576    657.5    48.63
5-1-y-q-a32-64-4-97-160      1048576    662.0    48.44
4-2-y-i-a-64-4-97-160        524288     343.5    47.23
4-2-y-k-a-32-4-97-160        524288     348.8    46.61
4-2-y-k-a-64-4-97-160        524288     358.5    46.18
4-1-y-k-a-32-4-97-160        524288     328.7    45.72
4-2-y-i-32768-64-4-97-160    524288     294.3    44.36
2-1-y-q-32768-32-4-97-160    262144     116.8    43.86
4-2-y-i-32768-32-4-97-160    524288     287.7    42.78
2-2-y-i-32768-32-4-97-160    262144     123.4    41.96
4-1-y-i-32768-64-4-97-160    524288     278.1    41.51
2-2-y-q-a-32-4-97-160        262144     151.5    41.14
4-2-y-i-a-32-4-97-160        524288     344.1    40.57
4-1-y-q-a-64-4-97-160        524288     336.2    40.42
2-2-y-i-a-32-4-97-160        262144     152.7    40.36
1-1-y-q-a-32-4-97-160        131072      70.5    40.20
4-1-y-k-32768-64-4-97-160    524288     274.3    39.88
2-2-y-k-32768-64-4-97-160    262144     132.6    37.13
2-2-y-k-a-64-4-97-160        262144     159.2    36.29
2-2-y-q-32768-64-4-97-160    262144     128.2    36.06
1-1-y-i-a-32-4-97-160        131072      66.6    32.71
5-1-y-q-a64-64-4-97-160      1048576    639.7    31.83
4-2-y-k-32768-64-4-97-160    524288     314.0    30.98
4-2-y-q-32768-64-4-97-160    524288     298.4    30.08
4-2-y-k-32768-32-4-97-160    524288     306.4    30.03
5-1-y-i-a64-32-4-97-160      1048576    628.3    29.63
5-1-y-q-a32-32-4-97-160      1048576    640.3    29.28
5-1-y-i-a64-64-4-97-160      1048576    643.0    28.95
4-2-y-q-a-64-4-97-160        524288     343.9    28.30
4-2-y-q-a-32-4-97-160        524288     345.6    28.25
```

### Observations by context

| Context | Configurations | Best decode | Worst decode | Spread |
|---|---|---|---|---|
| 128K | 24 | 65.5 t/s | 32.7 t/s | 2.0× |
| 256K | 24 | 72.9 t/s | 36.1 t/s | 2.0× |
| 512K | 24 | 66.3 t/s | 28.3 t/s | 2.3× |
| 1M | 8 | 52.1 t/s | 28.9 t/s | 1.8× |

**Parallel impact:** Every config with batch ≥2 shows a 30–60% decode penalty on 512K, and a **50–80% penalty on 1M** (see Parallel Tests).

---

## 🎯 NIAH Results (512K)

Needle-in-a-Haystack test with UUID needle at 10%, 50%, 90% depth. **Y** = found, **N** = not found.

| Run ID | KV | Batch | d10 | d50 | d90 | Score |
|---|---|---|---|---|---|---|
| `4-1-y-i-32768-32-4-97-160` | int8 | 1 | ✅ | ✅ | ✅ | **3/3** |
| `4-1-y-k-32768-32-4-97-160` | k8v4 | 1 | ✅ | ✅ | ✅ | **3/3** |
| `4-1-y-q-32768-32-4-97-160` | q4_0 | 1 | ✅ | ✅ | ✅ | **3/3** |
| `4-2-y-i-32768-32-4-97-160` | int8 | 2 | ✅ | ✅ | ✅ | **3/3** |
| `4-2-y-k-32768-32-4-97-160` | k8v4 | 2 | ✅ | ✅ | ✅ | **3/3** |
| `4-2-y-q-32768-32-4-97-160` | q4_0 | 2 | ✅ | ✅ | ✅ | **3/3** |

### Conclusion

- ✅ **KV quantization does not affect recall.** Even aggressive `q4_0` retains the needle at 50% depth — no "lost in the middle" failure.
- ✅ **Batch size does not affect recall.** Batch 2 scored identically to batch 1.
- ✅ **512K is genuinely functional** — not just accepted by the engine, but actively used by the model.

---

## ⚡ Parallel Tests

### Test 1 — Short prompts (~32K tokens)

**Setup:** 4 requests, 2 s stagger, `max_tokens=50`

| Config | Wall | Throughput | Avg TTFT | Avg decode |
|---|---|---|---|---|
| 256K batch 1 | 58.3 s | 3.4 t/s | — | — |
| **256K batch 4** | **43.2 s** | **4.6 t/s** | — | — |
| 512K batch 1 | 58.9 s | 3.4 t/s | — | — |
| **512K batch 4** | **43.7 s** | **4.6 t/s** | — | — |

**Result:** Batch 4 gives **+35% throughput** and **−26% wall time** at the cost of **+19% per-request latency**. ✅ Worth it.

### Test 2 — Long prompts (~64K tokens) on 1M context

**Setup:** 4 requests, 5 s stagger, `max_tokens=500`, `reasoning_budget_tokens=512`

| Config | Wall | Throughput | Avg TTFT | Avg decode | Expert cache |
|---|---|---|---|---|---|
| **1M batch 1** | **53.1 s** | **6.04 t/s** | **33.9 s** | **60.2 t/s** | **96–98%** |
| 1M batch 2 | 63.5 s | 5.34 t/s | 38.9 s | 14.8 t/s | ~85% |
| 1M batch 4 | 108.6 s | 3.18 t/s | 69.7 s | 10.0 t/s | 79–85% |

**Result:** Batch 2 and 4 are **catastrophic on 1M** — decode collapses from 60 t/s to 10–15 t/s.

### Why parallel fails at 1M

| Metric | Batch 1 | Batch 4 |
|---|---|---|
| Experts in VRAM | 7,260 | 2,682 |
| VRAM for expert cache | 13.77 GiB | 5.14 GiB |
| PCIe fallback | +2.7% | **+15.6%** |

**Root cause:** Each slot reserves its own 1M KV cache. With 4 sl
En el sitio

Enlaces a install, modelos, releases.