Issues / #1964
#1964 V0.1.43 Decode performance degradation on 4070ti + 14700 + 64 RAM, Linux IQ3_XXS
open · @Hotcooler · 5 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux
Beschreibung
Something changed in 43 that made decode on this setup to be about a third of what it was in 0.1.42.
Bench.sh output for v42:
```
Results (median, range over the runs):
chat decode tok/s 96.50 (80.10 - 108.00, n=9)
chat first token (s) 0.34 (0.30 - 0.39, n=9)
4K prompt tok/s 2181.30 (2143.90 - 2188.80, n=3)
4K first token (s) 1.86 (1.85 - 1.89, n=3)
16K prompt tok/s 3136.60 (3090.30 - 3139.70, n=3)
16K first token (s) 5.14 (5.14 - 5.22, n=3)
agent cache hit % 68.77 (68.70 - 68.80, n=3)
agent re-read tokens/session 4.00 (3.00 - 74.00, n=3)
agent turn first token (s) 1.94 (1.34 - 2.15, n=15)
agent decode tok/s 104.90 (73.20 - 131.40, n=15)
```
v43:
```
Results (median, range over the runs):
chat decode tok/s 37.90 (33.70 - 59.30, n=9)
chat first token (s) 0.37 (0.32 - 0.52, n=9)
4K prompt tok/s 2118.70 (2117.20 - 2173.80, n=3)
4K first token (s) 1.92 (1.87 - 1.92, n=3)
16K prompt tok/s 3140.70 (3095.00 - 3141.20, n=3)
16K first token (s) 5.14 (5.13 - 5.21, n=3)
agent cache hit % 68.73 (68.60 - 68.81, n=3)
agent re-read tokens/session 74.00 (4.00 - 74.00, n=3)
agent turn first token (s) 1.93 (1.33 - 2.01, n=15)
agent decode tok/s 45.90 (39.80 - 62.80, n=15)
```
"agent re-read tokens/session" jumps out. Though the range seems to be about the same.
Startup Json same for both:
```
{
"exe": "/opt/Servarr/llm/Strata/engine/strata",
"args": [
"--pack",
"/opt/Servarr/llm/Strata-data/packs/iq3_xxs",
"--native",
"/opt/Servarr/llm/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
"--ple-gguf",
"/opt/Servarr/llm/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf",
"--expert-profile",
"/opt/Servarr/llm/Strata/data/expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"auto:32768",
"--spec",
"4",
"--mtp",
"/opt/Servarr/llm/Strata-data/mtp/rt",
"--max-context",
"262144",
"--kv",
"int8",
"--kv-resident",
"32768",
"--vision",
"--vram-reserve-mib",
"600",
"--pool-affinity",
"auto",
"--pool-workers",
"15",
"--coupled-draft",
"--pcie-frac",
"0.24",
"--spec-min-p",
"0.70"
],
"cwd": "/opt/Servarr/llm/Strata",
"tokenizer": "/opt/Servarr/llm/Strata-data/packs/iq3_xxs/tokenizer",
"model_name": "qwen3.8-flash-next-iq3_xxs",
"log": "/opt/Servarr/llm/Strata/strata-iq3_xxs.log",
"lib_dirs": [
"/usr/local/cuda-13.0/bin",
"/usr/local/cuda-13.0/lib64"
],
"port": 8080,
"gpu": 0,
"gpus_asked": true,
"draft_vocab": "cyrillic",
"fit_max_tokens": true,
"expert_profile_save": "/opt/Servarr/llm/Strata-data/iq3_xxs-profile.bin",
"vision": {
"exe": "/opt/Servarr/llm/Strata/engine/strata-vision",
"mmproj": "/opt/Servarr/llm/Strata-data/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
"model": "/opt/Servarr/llm/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
"gpu": true,
"max_tokens": 3072
},
"sampling": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20,
"min_p": 0.0
}
}
```
Startup sh also the same for both:
```
#!/bin/sh
cd "/opt/Servarr/llm/Strata"
export STRATA_EXCHANGE_ROTATE=1
export STRATA_KV_PREFETCH=1
export STRATA_PREFILL_CPU_SHARE=auto
export STRATA_SPEC_COUPLED=1
export STRATA_SPEC_GUMBEL=1
exec "/opt/Servarr/llm/Strata/.venv/bin/python" "/opt/Servarr/llm/Strata/serve/server.py" "--engine" "strata" "--config" "/opt/Servarr/llm/Strata/strata-iq3_xxs.json" "--host" "0.0.0.0" "--port" "8080" "--api-key" "xxx" "--open"
```
Startup logs attached:
[v43.log](https://github.com/user-attachments/files/33306188/v43.log)
[v42.log](https://github.com/user-attachments/files/33306189/v42.log)
Suspected VRAM overflow with new changed, did not pan out, changing --vram-reserve-mib to 1000 did not produce better speeds.Mehr auf der Site
Links zu Install, Modellen, Releases.