Pull requests / #3274
#3274 [Benchmark] RTX 3080 Ti 12 GB in a KVM guest with Docker: 0.2.0-beta against 0.1.43
open · @moneymikeMD · 0 comentários · No GitHub
BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationLinux
Descrição
## Problem and result A results-only benchmark report for a card with no other report here: one RTX 3080 Ti 12 GB (sm_86) in a KVM guest with GPU passthrough, served from Docker. Re-run on 0.2.0-beta as asked when the tracker was reset; it replaces the 0.1.42 report this PR carried before. The 0.2.0-beta image (engine package 0.1.45.3) and the 0.1.43 image were run in alternation on the same model files at context 262,144, a fresh container per sweep, three runs per cell: - IQ3_S decode 76.4 / 77.1 / 68.4 tok/s on 0.1.43 against 97.0 / 92.2 / 81.8 on 0.2.0 at 4K / 32K / 128K (two sweeps each; every 0.2.0 sweep beat every 0.1.43 sweep). Prompt speed within 4%. With `STRATA_SM86_PERF=0` 0.2.0 gives 83.5 / 79.6 / 73.8, so the new sm_86 table is worth 14 / 14 / 10% here. `TUNE=quick`: "already fastest". - Swift 1.5 IQ3_S: 77.2 / 77.4 / 63.2 against 96.0 / 97.1 / 85.6. - UD-IQ4_XS: with 0.2.0's own settings the engine was killed for lack of RAM on this 94 GiB guest in 4 of 4 starts (kernel lines and engine log included). Reading `setup.py`, `unsloth_budget()` keeps every expert in RAM when the RAM is at least the experts plus 16 GiB, which does not count the PLE table locked in RAM or the KV streamed to RAM at this context; 0.1.43's setup wrote `--resident-budget-gib 55` for the same file. With that budget given by hand, 0.2.0 reads prompts 46 / 31 / 13% slower than 0.1.43 and decodes 21 / 7 / 11% slower; `STRATA_PREFILL_HOST_PLAN=0` changed nothing. - Needles 9 of 9 at 32K, 128K and 262K on 0.2.0 IQ3_S. ## Related discussion Replaces #2005, closed in the 0.2.0-beta tracker reset; `/reopen` passed its check but GitHub would not reopen it because the branch was re-pushed with the new report. No scope agreement needed (benchmark report). ## Approach New folder `bench/results/2026-10-19-community-rtx-3080-ti-kvm-docker-0.2.0/` (README, every run as JSON lines, the run configs with the key removed, the 0.2.0 engine logs, the scripts as run) and one entry in `docs/COMMUNITY_BENCHMARKS.md`. No code changes. ## Validation **Passed:** - `docker/build.sh --flavor cuda --engine <package> --tag strata:0.2.0-beta-cuda13 --base public.ecr.aws/docker/library/ubuntu:24.04` at v0.2.0-beta (3d1fb480) with the released `strata-engine-0.1.45.3-linux-x86_64-cuda-cuda13.tar.gz` (SHA-256 matches `SHA256SUMS.txt`), after adding `docs/TERMINAL_LAUNCH.md` to the script's `paths=` list (below); the build's own checks passed. - `sweep.py` (in the folder) and `tools/needle_bench.py` against both images on 2026-10-19, Ubuntu 26.04.1 guest, driver 615.71.09, Docker 29.9.0; the numbers above are from those runs (`sweep-*.jsonl`, `needles-iq3_s-0.2.0.json`). - The `STRATA_SET` values of each arm show up in that arm's engine log. **Failed:** - `docker/build.sh` unchanged at v0.2.0-beta: `missing payload item: docs/TERMINAL_LAUNCH.md` (its build-context list has `docs/MANAGEMENT_PROTOCOL.md` but not `docs/TERMINAL_LAUNCH.md`, which `packaging/package_terminal.py` requires; the line is the same on `main` at f57c0adc). Not fixed in this PR. - 0.2.0 UD-IQ4_XS with its own settings: OOM-killed, 4 of 4 starts, as above. **Not run:** - A bare-metal run of the same machine (VM and Docker cost not measured). - The 0.1.42 comparison is one sweep, after the others, not interleaved. - Generated tokens were not compared between versions; no quality check beyond the needles. - Vision, concurrent requests, a soak, agent sessions. ## Limits Three runs per cell and two sweeps per interleaved arm. The 0.1.43 and 0.1.42 images are sm_86-only source builds; the 0.2.0 image uses the released engine package. The cause of the slower UD-IQ4_XS prompt read with the budget was not found.
No site
Links install, modelos, releases.