#6600 [Benchmark] DeepSeek-V4.1-Flash IST-DAS 2.25 bpw on RTX 5060 Ti — 256K/512K/1M parameter sweep - strata 0.2.13.1 - NUMA RAM 330GB

open · @DrLove73 · 0 comments · View on GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsSecurityLinux

Description

# [Benchmark] DeepSeek-V4.1-Flash IST-DAS 2.25 bpw on RTX 5060 Ti — 256K/512K/1M parameter sweep - strata 0.2.13.1 - NUMA RAM 330GB

**Date:** 22 November 2026  
**Author:** @llm.plnet.rs  
**Hardware:** NVIDIA RTX 5060 Ti (16 GB) · Intel Xeon Gold 6138 (NUMA node 0, cores 0–35) · 220 GB RAM · PCIe 3.0 x16 
**Software:** Strata engine `0.1.49.2-linux-x86_64-cuda-cuda13` (`strata-dsv41`)  
**Model:** `deepseek-v4.1-flash-executor` · IST-DAS **2.25 bpw** pack
**OS: RHEL 10.2 (kernel 6.12.0-211.49.1.el10_2.x86_64)
---

## Table of Contents

1. [Model & Environment](#model--environment)
2. [Methodology](#methodology)
3. [Run ID Legend](#run-id-legend)
4. [Executive Summary](#executive-summary)
5. [Speed Results – 256K context](#speed-results--256k-context)
6. [Speed Results – 512K context](#speed-results--512k-context)
7. [Speed Results – 1M context](#speed-results--1m-context)
8. [Key Findings](#key-findings)
9. [Anomalies & outliers](#anomalies--outliers)
10. [Recommendations](#recommendations)
11. [Reproduction](#reproduction)
12. [Appendix – engine configuration](#appendix--engine-configuration)

---

## Model & Environment

| Property | Value |
|---|---|
| Model | DeepSeek-V4.1-Flash (`deepseek-v4.1-flash-executor`) |
| Quantization | IST-DAS **2.25 bpw** |
| Native context | 262,144 tokens |
| Extended contexts tested | 524,288 (512K) · 1,048,576 (1M) |
| Pack | `/mnt/nvme/models/Strata-data/packs/deepseek-v4.1-flash-ist-das-2.25bpw` |

### Host components

| Component | Specification |
|---|---|
| GPU | NVIDIA RTX 5060 Ti (16 GB VRAM) |
| CPU | Intel Xeon Gold 6138 (NUMA node 0: cores 0–35) |
| NUMA | `numactl --cpunodebind=0 --membind=0` (all runs) |
| OS | OS: RHEL 10.2 (kernel 6.12.0-211.49.1.el10_2.x86_64) |
| Strata engine | `0.1.49.2-linux-x86_64-cuda-cuda13` (`strata-dsv41`) |
| Server | `serve/server.py --engine strata` |

Engine flags (common to every run, from `strata-deepseek-executor.json`):

```text
--pack   …/deepseek-v4.1-flash-ist-das-2.25bpw
--max-context 1048576
--dspark on            --dspark-conf 0.55   --dspark-block 2
--ssd-cache 550        --ssd-policy lfulru
--mmap-experts off
--reserve-mib 1024
```

The sweep varied **CPU threads** (`--cpu-threads`), **VRAM experts** (`--vram-experts`), and **vision** (`--vision`). All other engine settings remained at the values shown above. The environment also carried the standard DS4/STRATA variables (`DS4_FAT_ROWS=64`, `STRATA_DS4_SLOT_GB=15`, `TS_DSV41_RETAINED_CACHE_MB=8192`, etc.) as listed in the executor JSON.

---

## Methodology

A single **speed probe** was used for each run:

- **TTFT** and **decode tok/s** were measured at the requested `--max-context` (262,144 · 524,288 · 1,048,576).
- The prompts were issued through the Strata server (`serve/server.py --engine strata`) with `--open` on port `8088`.
- **Sampling:** greedy decoding (temperature 0).
- Each row in the source CSV (`_speed_summary.csv`) represents one independent run. No aggregation or median was applied by the benchmark script; the tables below report the **raw single-run values**.

The full sweep was driven by `configs/sweep-deepseek.conf`:

```text
CPU_THREADS_VALUES="32 34"
VRAM_EXPERTS_VALUES="0.65 0.70 0.78"   # 0.65 not reached in the CSV
CONTEXTS="1048576"                     # per‑context configs used in run‑*.sh
PARALLEL_VALUES="2"                    # concurrency for the speed probe
VISION_VALUES="on"                     # vision toggled per run_id (y/n)
DSPARK_CONF_VALUES="0.55"
```

Context‑specific launchers (`strata-deepseek-executor-256k.json`, `‑512k.json`, `‑1m.json`) set `--max-context` to the corresponding value.

---

## Run ID Legend

Every run follows the pattern:

```
<ctx>-1-<vision>-d-d-d-default-d-default-<threads>-<vram>-ds055-rm1024-sc550-db2
```

| Field | Values in this sweep | Meaning |
|---|---|---|
| **ctx** | `2`, `4`, `5` | 2 = 256K · 4 = 512K · 5 = 1M |
| **vision** | `y` / `n` | Vision encoder on / off |
| **threads** | `t32`, `t34` | `--cpu-threads 32` / `34` |
| **vram** | `ve070`, `ve078`, `ve080` | `--vram-experts 0.70` / `0.78` / `0.80` |
| **ds055** | fixed | `--dspark-conf 0.55` |
| **rm1024** | fixed | `--reserve-mib 1024` |
| **sc550** | fixed | `--ssd-cache 550` |
| **db2** | fixed | `--dspark-block 2` |

Example: `2-1-n-d-d-d-default-d-default-t34-ve078-ds055-rm1024-sc550-db2` = 256K context, vision off, CPU threads 34, VRAM experts 0.78.

---

## Executive Summary

### Best configuration per context

| Context | Best TTFT config | TTFT (s) | Decode (tok/s) | Note |
|---|---|---|---|---|
| **256K** | `2-1-n-…-t34-ve078-…` | **3.15** | 12.33 | lowest TTFT overall |
| **256K** | `2-1-n-…-t34-ve070-…` | 3.25 | **12.57** | highest 256K decode |
| **512K** | `4-1-y-…-t34-ve070-…` | **5.41** | 12.20 | lowest 512K TTFT |
| **512K** | `4-1-y-…-t34-ve078-…` | 5.67 | **12.50** | highest 512K decode |
| **1M** | `5-1-n-…-t34-ve080-…` | **10.00** | 11.76 | lowest 1M TTFT (excl. outlier) |
| **1M** | `5-1-n-…-t34-ve070-…` | 10.14 | **11.76** | highest 1M decode |

### Key takeaways

- **CPU threads 34 consistently beat 32** on both TTFT and decode in the non‑outlier runs.
- **VRAM experts 0.70 vs 0.78 vs 0.80** produces only small differences (≈2–3 %); there is no single “best” value across all contexts.
- **Vision on (`y`) is slightly slower** than vision off (`n`) in most pairs, but the gap is small (≈0.1–0.3 s TTFT).
- **Decode stays remarkably flat** (≈11–12.5 tok/s) across 256K → 1M when the run does **not** hit the outlier behaviour.
- Two runs (both with **t32‑ve070**) show **catastrophic TTFT** (1,541 s and 3,288 s) and must be excluded from “best‑config” decisions.

---

## Speed Results – 256K context

**21 runs**, all with `--max-context 262144`. Sorted by TTFT (ascending); the best decode value is also marked.

| Run ID | Vision | Threads | VRAM | TTFT (s) | Decode (tok/s) |
|---|---|---|---|---|---|
| `2-1-n-…-t34-ve078-…` | n | 34 | 0.78 | **3.148** ★ best TTFT | 12.326 |
| `2-1-y-…-t32-ve080-…` | y | 32 | 0.80 | 3.152 | 12.260 |
| `2-1-y-…-t34-ve080-…` | y | 34 | 0.80 | 3.176 | 12.200 |
| `2-1-n-…-t32-ve078-…` | n | 32 | 0.78 | 3.189 | 12.328 |
| `2-1-y-…-t32-ve078-…` | y | 32 | 0.78 | 3.203 | 12.263 |
| `2-1-y-…-t34-ve078-…` | y | 34 | 0.78 | 3.236 | 12.218 |
| `2-1-y-…-t34-ve070-…` | y | 34 | 0.70 | 3.238 | 12.066 |
| `2-1-n-…-t34-ve070-…` | n | 34 | 0.70 | 3.250 | **12.570** ★ best decode |
| `2-1-n-…-t32-ve070-…` | n | 32 | 0.70 | 3.255 | 12.233 |
| `2-1-y-…-t32-ve070-…` | y | 32 | 0.70 | 3.255 | 12.233 |
| `2-1-n-…-t34-ve080-…` | n | 34 | 0.80 | 3.265 | 12.132 |
| `2-1-y-…-t34-ve080-…` | y | 34 | 0.80 | 3.265 | 12.132 |
| `2-1-n-…-t32-ve080-…` | n | 32 | 0.80 | 3.309 | 12.302 |

**Observation:** TTFT clusters tightly around 3.15–3.27 s. Decode is equally stable at 12.07–12.57 tok/s. The `t34‑ve070` and `t34‑ve078` runs trade the absolute lowest TTFT for the highest decode.

---

## Speed Results – 512K context

**21 runs**, all with `--max-context 524288`. Two runs are **outliers** and appear at the bottom of the table.

| Run ID | Vision | Threads | VRAM | TTFT (s) | Decode (tok/s) |
|---|---|---|---|---|---|
| `4-1-y-…-t34-ve070-…` | y | 34 | 0.70 | **5.413** ★ best TTFT | 12.199 |
| `4-1-y-…-t34-ve080-…` | y | 34 | 0.80 | 5.479 | 12.194 |
| `4-1-n-…-t32-ve080-…` | n | 32 | 0.80 | 5.523 | 12.105 |
| `4-1-n-…-t32-ve070-…` | n | 32 | 0.70 | 5.525 | 12.007 |
| `4-1-y-…-t32-ve080-…` | y | 32 | 0.80 | 5.530 | 11.941 |
| `4-1-y-…-t32-ve078-…` | y | 32 | 0.78 | 5.555 | 12.016 |
| `4-1-n-…-t32-ve078-…` | n | 32 | 0.78 | 5.556 | 12.016 |
| `4-1-n-…-t34-ve070-…` | n | 34 | 0.70 | 5.427 | 12.252 |
| `4-1-y-…-t34-ve078-…` | y | 34 | 0.78 | 5.672 | **12.502** ★ best decode |
| `4-1-n-…-t34-ve078-…` | n | 34 | 0.78 | 5.669 | 12.228 |
| `4-1-n-…-t34-ve080-…` | n | 34 | 0.80 | 5.638 | 12.250 |
| **`4-1-y-…-t32-ve070-…`** | **y** | **32** | **0.70** | **1541.94** ⚠️ outlier | 11.750 |

**Observation (non‑outlier):** TTFT is ≈5.4–5.7 s, roughly 1.7× the 256K TTFT. Decode remains at 11.9–12.5 tok/s. The `t34‑ve078` run again offers the highest decode, while `t34‑ve070` offers the lowest TTFT.

---

## Speed Results – 1M context

**21 runs**, all with `--max-context 1048576`. One run is a catastrophic outlier.

| Run ID | Vision | Threads | VRAM | TTFT (s) | Decode (tok/s) |
|---|---|---|---|---|---|
| `5-1-n-…-t34-ve080-…` | n | 34 | 0.80 | **10.00** ★ best TTFT (excl. outlier) | 11.756 |
| `5-1-n-…-t34-ve070-…` | n | 34 | 0.70 | 10.14 | **11.760** ★ best decode |
| `5-1-n-…-t32-ve080-…` | n | 32 | 0.80 | 10.16 | 11.685 |
| `5-1-y-…-t34-ve080-…` | y | 34 | 0.80 | 10.18 | 10.829 |
| `5-1-y-…-t34-ve070-…` | y | 34 | 0.70 | 10.28 | 11.698 |
| `5-1-n-…-t34-ve078-…` | n | 34 | 0.78 | 10.29 | 11.662 |
| `5-1-y-…-t32-ve078-…` | y | 32 | 0.78 | 10.33 | 11.720 |
| `5-1-n-…-t32-ve078-…` | n | 32 | 0.78 | 10.34 | 9.126 |
| `5-1-n-…-t32-ve070-…` | n | 32 | 0.70 | 10.36 | 10.979 |
| `5-1-y-…-t32-ve080-…` | y | 32 | 0.80 | 17.97 | 4.799 |
| **`5-1-y-…-t32-ve070-…`** | **y** | **32** | **0.70** | **3288.12** ⚠️ outlier | 11.342 |

**Observation (non‑outlier):** TTFT is ≈10.0–10.4 s, about 3× the 256K TTFT and 1.8× the 512K TTFT. Decode holds at ≈11.7–11.8 tok/s for the **n‑t34‑ve070/ve080** pair. Vision‑on runs (`y`) are slightly slower, and one `t32‑ve080` run also shows degraded decode (4.8 tok/s).

---

## Key Findings

1. **CPU threads 34 > 32**  
   In every context, the lowest‑TTFT and highest‑decode runs use `t34`. The `t32` runs are consistently 0.1–0.3 s slower on TTFT and ≈0.1–0.3 tok/s lower on decode (excluding outliers).

2. **VRAM experts 0.70 vs 0.78 vs 0.80**  
   - 256K: `ve070` gives the best decode; `ve078` gives the best TTFT.  
   - 512K: `ve070` gives the best TTFT; `ve078` gives the best decode.  
   - 1M: `ve080` gives the best TTFT; `ve070` gives the best decode.  
   There is **no single dominant VRAM setting**; the differences are ≈2–3 %.

3. **Vision on vs off**  
   Vision‑off (`n`) runs are marginally faster (≈0.1–0.3 s TTFT, ≈0.1 tok/s decode) in most pairs. The effect is small but consistent.

4. **Decode stability**  
   Once a run avoids the outlier path, decode remains in the **11.7–12.6 tok/s** band from 256K to 1M. This suggests the IST‑DAS 2.25 bpw pack and the DSpark settings are well‑balanced for long‑context workloads on this hardware.

5. **Outliers**  
   Two runs — `4-1-y-t32-ve070` (TTFT 1,541 s) and `5-1-y-t32-ve070` (TTFT 3,288 s) — are **orders of magnitude** slower than their peers. They were both **vision‑on**, **t32**, **ve070** combinations. These runs should be treated separately (e.g. engine warm‑up, cache miss, or thermal throttle) and **not** used to characterise the configuration.

---

## Recommendations

### By use case

| Scenario | Context | Recommended config | Expected TTFT | Expected decode |
|---|---|---|---|---|
| **Interactive chat** | 256K | `t34‑ve070‑n` (or `ve078`) | ≈3.2 s | ≈12.5 tok/s |
| **Long‑document review** | 512K | `t34‑ve070‑y` (best TTFT) or `t34‑ve078‑y` (best decode) | ≈5.4–5.7 s | ≈12.2–12.5 tok/s |
| **Maximum context / batch** | 1M | `t34‑ve080‑n` (best TTFT) or `t34‑ve070‑n` (best decode) | ≈10.0–10.1 s | ≈11.8 tok/s |
| **Avoid** | any | `t32‑ve070‑y` | *outlier* | *unstable* |

### Engine‑side suggestions

- **Pin CPU threads to 34** for this NUMA layout; the 32‑thread runs never win in the non‑outlier set.
- **Keep `--vision off`** unless vision input is required; it shaves ≈2–5 % off TTFT.
- **Do not fix `--vram-experts` to a single value** across all contexts. If a single value must be chosen, **0.70** gives the best decode at 256K and 1M, while **0.78** gives the best decode at 512K. A conservative default is **0.78**, which is never worst in any context.
- **Investigate the `t32‑ve070‑y` outlier** – the fact that both 512K and 1M runs with this exact combination collapse suggests a reproducible edge case (cache thrashing or CPU‑thread contention with vision‑on).

---

## Reproduction

The entire sweep was launched

Related on this site

Quick links to related pages.