#3258 Strata Benchmark Report — Qwen3.8 Flash Next (Unsloth UD-Q4_K_XL) on 0.1.45, with --vision=cpu, 260+ kombinations

Description

# Strata Benchmark Report — Qwen3.8 Flash Next (Unsloth UD-Q4_K_XL) on 0.1.45, with --vision=cpu, 260+ kombinations

**Model family:** Qwen3.8-Flash-Next  
**Quantization:** Unsloth Dynamic UD-Q4_K_XL (4-bit, K-quant experts)  
**Engine:** Strata 0.1.45 (llama.cpp 3cf0325)  
**Hardware:** RTX PRO 4000 Blackwell (24 GB VRAM), Intel Xeon Gold 6138, 220 GB RAM  
**Dates:** 2026-10-17 (Legacy), 2026-10-19 (Strata 0.1.45 `vision=cpu` re-run)  
**Contexts:** 128K, 256K, 384K, 512K  
**Total runs:** 200+

## Summary

Peak decode is **42.70 tok/s at 256K**. Unlike the Swift IQ3_S model, offloading the vision encoder to the CPU (`vision=cpu`) **did not improve performance** for UD-Q4_K_XL and actually slightly degraded it at 256K (39.30 vs 42.70 tok/s). This is likely because the 111 GB model footprint leaves the expert cache severely VRAM-starved regardless of the ~1.5 GB saved from the vision encoder. However, performance remains remarkably stable at ~35-40 tok/s across all contexts up to 512K.

## Sweep configuration

```
CONTEXTS         = 131072 262144 393216 524288
PARALLEL_VALUES  = 1 2 4
VISION_VALUES    = cpu / off / on
KV_VALUES        = int8 q4_0 k8v4
PREFILL_VALUES   = 32768 65536 auto
KV_RESIDENT      = 32768 65536
SPEC             = 4
ADAPT_DECAY      = 0.97
ADAPT_SWAPS      = 160
```

## Results — Best config per context

| Context | Config (kv, prefill, kvr, vision) | TTFT | Decode | run_id |
|---|---|---|---|---|
| 128K | q4_0, 32768, 32768, cpu | 56.9 s | **40.20 tok/s** | 1-1-c-q-32-32-4-97-160 |
| 256K | q4_0, 32768, 32768, off | 124.1 s | **42.70 tok/s** | 2-1-n-q-32-32-4-97-160 |
| 384K | k8v4, auto, 32768, off | 303.2 s | **38.92 tok/s** | 3-1-n-k-a-32-4-97-160 |
| 512K | int8, auto, 32768, off | 417.3 s | **37.50 tok/s** | 4-1-n-i-a-32-4-97-160 |

## Vision Mode Impact (The anomaly)

While `vision=cpu` helped Swift, it slightly hurts Q4_K_XL due to the sheer size of the model's experts.

| Context | Best Config (kv, prefill) | vision=off | vision=cpu | Impact |
|---|---|---|---|---|
| 128K | q4_0, 32768 | 38.80 | **40.20** | +3.6% |
| 256K | int8, 32768 | **42.70** *(q4_0)* | 39.30 | -8.0% |
| 384K | q4_0, 65536 | **38.65** | 36.39 | -5.8% |
| 512K | q4_0, 32768 | **37.23** | 36.66 | -1.5% |

*Note: The VRAM savings from the CPU encoder are not enough to unlock a new tier of expert caching for a 111 GB model on a 24 GB GPU.*

## Key findings

1. **~42 tok/s peak** — Q4_K_XL is roughly half the decode speed of Swift IQ3_S at equivalent context. Expected: more bits per expert, more RAM pressure.
2. **256K is the sweet spot**, but performance is essentially flat (37–43 tok/s) from 128K to 512K.
3. **KV type shifts by context**: 
   - 128K–256K: `q4_0` (best balance of memory and speed).
   - 384K+: `k8v4` or `int8` (better streaming when VRAM is tight).
4. **prefill=32768** wins at 128K–256K; `auto` wins at 384K–512K.
5. **Parallelism hurts badly** — `batch=2` drops speed by ~20-30%, and `batch=4` by up to 45% (e.g., 42.70 down to 24.19 at 256K).
6. **Quality Check needed**: Unlike Swift, we do not have NIAH quality results for Unsloth. This should be added to verify if Q4_K_XL's higher precision justifies the ~50% speed penalty.

## Recommended production configs

| Use case | Config |
|---|---|
| Chat (128K) | `kv=q4_0, prefill=32768, kvr=32768, vision=cpu` |
| Long analysis (256K) | `kv=q4_0, prefill=32768, kvr=32768, vision=off` |
| Document QA (512K) | `kv=int8, prefill=auto, kvr=32768, vision=off` |

### Representative Results & Speed Degradation Analysis (Q4_K_XL)

| Scenario / Purpose | Date | Context | KV / Prefill / KVR | TTFT (s) | Decode (tok/s) | run_id |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **128K - Best CPU Vision** | Oct 19 | 131,072 | `q4_0` / `32768` / `32768` | 56.9 | **40.20** | 1-1-c-q-32-32-4-97-160 |
| 128K - Best OFF Vision | Oct 19 | 131,072 | `q4_0` / `32768` / `32768` | 57.9 | 38.80 | 1-1-n-q-32-32-4-97-160 |
| 128K - Best ON Vision | Oct 17 | 131,072 | `int8` / `auto` / `32768` | 93.3 | 35.79 | 1-1-n-i-a-32-4-97-160 |
| 128K - Batch 2 Penalty | Oct 17 | 131,072 | `int8` / `32768` / `32768` | 57.9 | 28.89 | 1-2-n-i-32-32-4-97-160 |
| 128K - Batch 4 Penalty | Oct 17 | 131,072 | `k8v4` / `32768` / `32768` | 69.8 | 22.64 | 1-4-n-k-32-32-4-97-160 |
| **256K - Absolute Peak (OFF)** | Oct 17 | 262,144 | `q4_0` / `32768` / `32768` | 124.1 | **42.70** | 2-1-n-q-32-32-4-97-160 |
| 256K - Best CPU Vision | Oct 19 | 262,144 | `int8` / `32768` / `65536` | 110.5 | 39.30 | 2-1-c-i-32-64-4-97-160 |
| 256K - CPU Vision (`q4_0`) | Oct 19 | 262,144 | `q4_0` / `32768` / `32768` | 124.8 | 38.92 | 2-1-c-q-32-32-4-97-160 |
| 256K - Best ON Vision | Oct 17 | 262,144 | `int8` / `32768` / `32768` | 109.9 | 38.23 | 2-1-y-i-32-32-4-97-160 |
| 256K - Batch 2 Penalty | Oct 17 | 262,144 | `q4_0` / `32768` / `65536` | 140.4 | 34.49 | 2-2-n-q-32-64-4-97-160 |
| 256K - Batch 4 Penalty | Oct 17 | 262,144 | `q4_0` / `32768` / `32768` | 145.6 | 24.19 | 2-4-n-q-32-32-4-97-160 |
| **384K - Best OFF Vision** | Oct 18 | 393,216 | `k8v4` / `auto` / `32768` | 303.2 | **38.92** | 3-1-n-k-a-32-4-97-160 |
| 384K - Best CPU Vision | Oct 19 | 393,216 | `q4_0` / `65536` / `65536` | 193.2 | 36.39 | 3-1-c-q-64-64-4-97-160 |
| 384K - CPU Vision (`q4_0`) | Oct 19 | 393,216 | `q4_0` / `32768` / `65536` | 182.4 | 36.07 | 3-1-c-q-32-64-4-97-160 |
| 384K - Best ON Vision | Oct 18 | 393,216 | `k8v4` / `32768` / `32768` | 213.1 | 34.68 | 3-1-y-k-32-32-4-97-160 |
| **512K - Best OFF Vision** | Oct 18 | 524,288 | `int8` / `auto` / `32768` | 417.3 | **37.50** | 4-1-n-i-a-32-4-97-160 |
| 512K - Best CPU Vision | Oct 19 | 524,288 | `q4_0` / `65536` / `32768` | 262.4 | 36.66 | 4-1-c-q-64-32-4-97-160 |
| 512K - CPU Vision (`q4_0`) | Oct 19 | 524,288 | `q4_0` / `32768` / `32768` | 283.4 | 36.05 | 4-1-c-q-32-32-4-97-160 |
| 512K - Best ON Vision | Oct 18 | 524,288 | `q4_0` / `auto` / `32768` | 424.8 | 35.89 | 4-1-y-q-a-32-4-97-160 |
| 512K - Batch 2 Penalty | Oct 18 | 524,288 | `int8` / `auto` / `65536` | 519.6 | 24.62 | 4-1-n-q-a-64-4-97-160 |

#### Key Takeaways from Representative Data (Q4_K_XL):
*   **Vision CPU vs OFF Anomaly:** Unlike Swift IQ3_S, `vision=cpu` does *not* yield a massive speedup for this model. The 111 GB footprint means experts are starved for VRAM regardless. At 256K, `vision=off` actually wins (42.70 vs 38.92 tok/s).
*   **Severe Batch Penalties:** Adding parallel slots destroys throughput. At 256K, `batch=4` slashes decode speed by almost half (42.70 down to 24.19 tok/s).
*   **KV Cache Scaling:** `q4_0` is optimal for 128K-256K. At 384K+, `k8v4` and `int8` take over, likely due to more efficient streaming when VRAM is pushed to the absolute limit.