Pull requests / #2066

#2066 bench: dual Quadro RTX 6000 results, upgrade A/B and NUMA comparison

open · @ambu50 · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsDocumentation

Beschreibung

## Summary

Adds a results-only community report for a dual Quadro RTX 6000 24 GiB / dual
Xeon Gold 6138 ThinkStation P720 running Unsloth UD-Q4_K_XL.

- Official `tools/strata_bench.py` suite on v0.1.43, three repetitions and `--long`.
- Standard-suite medians: chat decode 65.7 tok/s, agent follow-up decode 70.5 tok/s;
  fresh 4K/16K/32K prefill 840.8/1542.0/1796.7 tok/s.
- Supplementary old/new/new/old comparison: median short decode 66.95 -> 74.80
  tok/s (+11.7%) from 0.1.40.3 to 0.1.43; prefill essentially unchanged.
- Supplementary NUMA experiment on 0.1.40.3 after expanding GPU-local socket RAM:
  interleave outperformed node-0-only memory placement on the tested corpus.
- Raw per-request records, timing excerpts, verified placement summaries,
  sanitized config, custom corpus and reproduction instructions included.

## Scope and limitations

The standard suite and custom comparisons are reported separately. Two launches
per custom configuration/version; three repeats in the standard suite. Configured
context is 262144, largest tested input is 43440 tokens. No long-context recall or
quality evaluation. The version comparison uses release defaults, including
STRATA_ROUTE_TAIL_SKIP=7, which can change answers. No claim of byte-identical
output or causal attribution of speedups. CPU vision helper was retained.

The official report's auto-detected CUDA 13.2 field is explicitly identified as
driver capability; the engine was built with CUDA toolkit 12.4 for sm_75.

No engine changes or model files are included. Adds a report link to
docs/COMMUNITY_BENCHMARKS.md.

Related measurement context: #1900 (opt-in NUMA placement proposal). This report
provides workload-specific measurements; it does not implement that proposal or
claim that GPU-local placement always helps.

## Validation

- Official suite completed; Hermes restored and Strata healthy.
- All JSON artifacts parse.
- Personal paths and unrelated live-user traffic excluded from packaged data.
- `git diff --cached --check` passes.

Mehr auf der Site

Links zu Install, Modellen, Releases.