Issues / #494
#494 Decode profile (RTX A3000 + i7-12850HX): the CPU expert pool is ~half the round; pin the verify host first
closed · @yannickloth · 4 comments · View on GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
Description
## Context A decode-time profile of the RTX A3000 12 GB laptop + i7-12850HX host from #485, on production engine **0.1.35** (Qwen3.8-Flash-Next Swift 1.5 IQ3_XXS native pack, the calibrated `--pcie-frac 0.35 --spec-min-p 0.70 --pool-workers 8`), from the engine's own `--stats` accumulators. Full write-up and raw lines: `bench/results/2026-10-02-rtx-a3000-i7-12850hx/profile.md` (PR #489). ## The split One verify-window round, ms and share of the round's wall clock. A round emits 2.10 tokens at the fresh hit rate and 1.15 when the output degenerates and hits more. | Term | fresh decode, `hit_rate 0.354` | ms | warm/degenerate, `hit_rate 0.756` | ms | | --- | --- | ---: | --- | ---: | | GPU — "wait for rings" | 34% | 26.7 | 62% | 23.1 | | CPU expert pool | **53%** | 41.6 | 26% | 9.6 | | MTP drafter | 4% | 3.1 | 6% | 2.1 | | host | 2% | 1.7 | 1% | 0.2 | | verify copy-stream commit | ~0% | 0.03 | ~0% | 0.03 | | embed/sample/round overhead | 6% | 4.9 | 6% | 2.2 | | round | | 78.1 (26.9 tok/s) | | 37.3 (30.9 tok/s) | Real serve traffic on this host ran `hit_rate 0.35-0.65` (34-40 tok/s), i.e. the fresh-decode end. There the **CPU expert pool is the largest single term**, and the GPU is second. ## What it sizes - **Pinning the verify host thread is the top host-side lever.** The pool is 26-53% of the round; recovering the `pool.hpp` pinned vs in-loop rates (36.3 vs 26.9 GB/s, ~1.35×) is worth ~+8-15% tok/s. The pool measures only **18-22 GB/s over the rows** here, so there may be more than 1.35× to win. The `generate`/session path already pins the host (`SessionLoopScratch::init`), but the serve/spec `Verifier::run` does not — the doorbell-spinning host and the PCIe-copy helpers are free to land on a worker's core. - **256-bit AVX-VNNI is bounded.** The down path is ~29% of the pool; a 1.5-2× down path is ~5-8% of the round. The i-quant gate/up is ~68% of the pool and documented codebook-bound, so VNNI barely touches it. - **The per-window copy-stream `cudaStreamSynchronize` is already ~0 ms/round on this host.** The MTP drafter's per-draft syncs live inside the 4-6% MTP term and are the only part of that area with headroom. - **P/E-aware division:** `--pool-workers 8` on the default `--pool-affinity all` does **not** land on 8 P-cores; it pins cores `2,4,6,8,10,12,14,16` = seven P-core primaries + the first E-core, and the E-core is the tail of every split-row drain. `--pool-affinity p-cores` avoids it but starts only `p_cores - 1 = 7` workers. ## Notes for measuring this model The native IQ pack runs verify windows only, so `--gpu-stages`, `--gpu-only-full` and `--no-pool`+`--spec` are unavailable (`session_replay_stages: not captured with the split`, etc.), and `/metrics`' `ram_blobs`/`file_blobs`/`file_mb` are 0 on the `MAP_HUGETLB` arena. The verify-window "wait for rings" term is the GPU floor available on this path; the 26.32 ms graph floor is the non-native path's. Refs #485 (the probe's low reading matters via `pcie_frac`, not via bandwidth: PCIe in use is ~4 GB/s) and #487.
Related on this site
Quick links to related pages.