Pull requests / #2005

#2005 Community benchmark: RTX 3080 Ti 12 GB, KVM guest with GPU passthrough, Docker (engine 0.1.42)

open · @moneymikeMD · 0 commentaires · Sur GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationLinux

Description

Adds `bench/results/2026-10-11-community-rtx-3080-ti-kvm-docker/`. Results only; no engine changes.

**Hardware.** One RTX 3080 Ti 12 GB (PCIe Gen 4 x16) in a Linux KVM guest with GPU passthrough: 32 vCPUs of a Xeon w7-2475X, 96 GiB of DDR5-4800, Ubuntu 26.04. The engine is built and run from this repository's Dockerfile at v0.1.42.

**Models.** IQ3_S, UD-IQ4_XS and Swift 1.5 IQ3_S, context 262,144, thinking off.

**What was measured.**
- The 4K / 32K / 128K sweep, 256 generated tokens, three runs per cell, with setup's defaults for all three models and after `setup.py --calibrate` for IQ3_S and UD-IQ4_XS.
- `tools/needle_bench.py` at 32k, 128k and 262k, depths 10, 50 and 90: 9 of 9.
- A context ladder from 131,072 to 524,288, batch slots (`"parallel": 2`) on a 12 GB card, and agent sessions with and without the conversation cache.

**Headline figures (median).** IQ3_S decodes at 93.9 / 81.7 / 75.2 tok/s and reads prompts at 1,807 / 2,274 / 2,180 tok/s at 4K / 32K / 128K. Calibration changes nothing for IQ3_S here and raises UD-IQ4_XS decode from 43.7 / 39.8 / 37.6 to 54.9 / 53.0 / 55.5 tok/s, mostly through 21 pool workers instead of the default 31.

**Limitations.**
- No bare-metal run of the same machine, so nothing here measures what the virtual machine or Docker cost.
- Three runs per cell. The context ladder and the batch-slot section come from an earlier script with its own prompt sizes; the README marks them.
- The agent section is a timing and caching observation on one private task, not an agent benchmark.
- Setup reported the Swift 1.5 shards as no longer at the pinned revision.
- Measured on engine 0.1.42.

Sur le site

Liens install, modèles, releases.