Issues / #494

#494 Decode profile (RTX A3000 + i7-12850HX): the CPU expert pool is ~half the round; pin the verify host first

closed · @yannickloth · 4 commentaires · Sur GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows

Description

## Context

A decode-time profile of the RTX A3000 12 GB laptop + i7-12850HX host from #485, on production engine **0.1.35**
(Qwen3.8-Flash-Next Swift 1.5 IQ3_XXS native pack, the calibrated `--pcie-frac 0.35 --spec-min-p 0.70
--pool-workers 8`), from the engine's own `--stats` accumulators. Full write-up and raw lines:
`bench/results/2026-10-02-rtx-a3000-i7-12850hx/profile.md` (PR #489).

## The split

One verify-window round, ms and share of the round's wall clock. A round emits 2.10 tokens at the fresh hit
rate and 1.15 when the output degenerates and hits more.

| Term | fresh decode, `hit_rate 0.354` | ms | warm/degenerate, `hit_rate 0.756` | ms |
| --- | --- | ---: | --- | ---: |
| GPU — "wait for rings" | 34% | 26.7 | 62% | 23.1 |
| CPU expert pool | **53%** | 41.6 | 26% | 9.6 |
| MTP drafter | 4% | 3.1 | 6% | 2.1 |
| host | 2% | 1.7 | 1% | 0.2 |
| verify copy-stream commit | ~0% | 0.03 | ~0% | 0.03 |
| embed/sample/round overhead | 6% | 4.9 | 6% | 2.2 |
| round | | 78.1 (26.9 tok/s) | | 37.3 (30.9 tok/s) |

Real serve traffic on this host ran `hit_rate 0.35-0.65` (34-40 tok/s), i.e. the fresh-decode end. There the
**CPU expert pool is the largest single term**, and the GPU is second.

## What it sizes

- **Pinning the verify host thread is the top host-side lever.** The pool is 26-53% of the round; recovering the
  `pool.hpp` pinned vs in-loop rates (36.3 vs 26.9 GB/s, ~1.35×) is worth ~+8-15% tok/s. The pool measures only
  **18-22 GB/s over the rows** here, so there may be more than 1.35× to win. The `generate`/session path already
  pins the host (`SessionLoopScratch::init`), but the serve/spec `Verifier::run` does not — the doorbell-spinning
  host and the PCIe-copy helpers are free to land on a worker's core.
- **256-bit AVX-VNNI is bounded.** The down path is ~29% of the pool; a 1.5-2× down path is ~5-8% of the round.
  The i-quant gate/up is ~68% of the pool and documented codebook-bound, so VNNI barely touches it.
- **The per-window copy-stream `cudaStreamSynchronize` is already ~0 ms/round on this host.** The MTP drafter's
  per-draft syncs live inside the 4-6% MTP term and are the only part of that area with headroom.
- **P/E-aware division:** `--pool-workers 8` on the default `--pool-affinity all` does **not** land on 8 P-cores;
  it pins cores `2,4,6,8,10,12,14,16` = seven P-core primaries + the first E-core, and the E-core is the tail of
  every split-row drain. `--pool-affinity p-cores` avoids it but starts only `p_cores - 1 = 7` workers.

## Notes for measuring this model

The native IQ pack runs verify windows only, so `--gpu-stages`, `--gpu-only-full` and `--no-pool`+`--spec` are
unavailable (`session_replay_stages: not captured with the split`, etc.), and `/metrics`'
`ram_blobs`/`file_blobs`/`file_mb` are 0 on the `MAP_HUGETLB` arena. The verify-window "wait for rings" term is
the GPU floor available on this path; the 26.32 ms graph floor is the non-native path's.

Refs #485 (the probe's low reading matters via `pcie_frac`, not via bandwidth: PCIe in use is ~4 GB/s) and #487.

Sur le site

Liens install, modèles, releases.