反馈 / #1863
#1863 fix: Complete the synchronous decode timing breakdown
open · @SanghunYun · 0 评论 · 去 GitHub 看
Server & APIAMD / HIPNVIDIA / CUDADocumentationWindows
说明
## What do you want to do? Include expert swap application and synchronous adaptive-worker join time in the serial `--serve` breakdown printed by `STRATA_DECODE_TIMING=1`. The total already includes these intervals, but `verify + commit/emit + draft` does not, which makes the decode time harder to interpret. ### Observed on v0.1.42 I sent the same public request in release → instrumented build → release order, using v0.1.42 (`61b3fb5d`). The two unmodified release runs showed: | Run | Total decode (ms/window) | Existing sections: verify + commit/emit + draft | Time outside the breakdown | |---|---:|---:|---:| | Release before | 53.54 | 38.69 + 0.05 + 2.88 = 41.62 | **11.92** | | Release after | 54.03 | 38.97 + 0.05 + 2.85 = 41.87 | **12.16** | The values inside the verify parentheses describe verify itself; they are not added again. ### Intervals to include | Order | Interval | Code location | |---|---|---| | Before verify | Apply the expert swaps prepared by the previous window | The `apply_pending()` branch | | After draft | Wait for the synchronous adaptive worker to finish | `adapt_thr.join()` | Direct timers around these locations recorded **5.37ms/window for swap application** and **6.91ms/window for join waiting**. Together with the existing sections, they accounted for that run's total of 53.76ms/window. Swap application includes GPU completion synchronization, RAM exchange commit and residency updates. The join timer measures the caller's wait, not the worker's whole parallel runtime. My current workaround is to subtract the three sections from the total. Showing these intervals in execution order on the same line would make the breakdown easier to read directly from the log. ### Reproduction conditions - v0.1.42 (`61b3fb5d`), CUDA sm_86 source build; RTX 3060 12GB, 32GB system RAM, Ubuntu 26.04. - Qwen3.8 Flash-Next IQ2_XS; 3,196 actual VRAM expert slots, 20.5GiB resident RAM, KV `k8v4`, `--spec 3 --mtp-max-t 3 --spec-min-p 0.5`. - `STRATA_DECODE_TIMING=1`, synchronous adaptation (`--adapt-async` unset). Fresh process and the same static expert profile per run; profile saving, `STRATA_RAM_ADAPT` and GPU verify profiling off. `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_IQ_MT_MIN=1`. - The same request containing the first 12,000 characters of public `docs/DETAILS.md`, sent through the upstream Python server: 4,489 input tokens, 256 output tokens, 115 windows. Zero reused prompt tokens; OS page cache not flushed. The accompanying PR includes the request helper and the timing changes. The instrumented build was run on CUDA and also compiled and linked as HIP Release. AMD GPU inference was not tested. <details> <summary>Original release timing line</summary> ```text strata decode timing: 115 windows, avg T 2.51, 2.23 tokens/window, 53.54 ms/window = verify 38.69 (GPU-reach wait 19.69 + per-layer host 17.40 [plan 0.16 actq 0.13 jobs 2.53 CPU 14.54] + stage 0.01) + commit/emit 0.05 + draft 2.88; per layer-window: CPU experts 4.65 (5.96 entries), VRAM hits 15.95, PCIe 0.39 ``` </details>
本站相关内容
相关页面的快捷入口。