改动 / #2673
#2673 native dense: upload the projection matrices from 8 threads through pinned staging (1.5 s -> 0.3 s)
open · @page1 · 0 评论 · 去 GitHub 看
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
说明
## Title Issue: none ## Summary On Windows the shard mapping is not read ahead, so one synchronous `cudaMemcpy` per matrix took its page faults one at a time: 300 projection matrices, 2019 MiB, in 1.5 s. Now 8 threads copy them in 8 MiB pieces through pinned staging: 0.3 s. The GPU holds the same bytes (checksum below). ## What changed - `src/core/native_dense.cpp`: the matrices are allocated as before, then copied in 8 MiB pieces by 8 threads. On Windows each thread reads its piece with `ReadFile` into a double-buffered pinned staging buffer and `cudaMemcpyAsync` on its own stream. Elsewhere, or if the file cannot be opened, a piece is copied from the mapping. - `STRATA_DENSE_UPLOAD_THREADS=N` sets the threads; `1` keeps the serial path. - `STRATA_DENSE_CHECKSUM=1` hashes the matrices as the GPU holds them: `e2a34e261e2077dd` for both the threaded and the serial path (IQ3_S, 300 matrices, on v0.1.44.1). ## Extra Notes Measured on an RTX 3090 24 GB, Core Ultra 9 285K, 95 GB DDR5, WD SN8100, Windows 11 26300, IQ3_S at 262K context, against a source build of v0.1.44.1 (`296b59d2`) the same day. 6 + 6 server starts in a rotated order, `/health` polled every 0.25 s: | | v0.1.44.1 | this branch | |---|---|---| | projection matrices (the engine's own line) | 1.5 s (every start) | **0.3 s** (every start) | | server start to loaded, median (range) | 17.82 s (16.91-18.23) | 16.92 s (16.43-17.13) | | decode at 1.7K / 61K tokens (3 + 3 starts) | 89.2 / 85.4 tok/s | 89.5 / 84.9 tok/s | | prompt at 61K tokens | 2718 tok/s | 2699 tok/s | Built: CUDA 13.4 (sm_86) only. **Not built for HIP or SYCL** (no ROCm or oneAPI here), and the non-Windows path (from the mapping) was not run. `native_dense.cpp` is part of the HIP build, so it needs a HIP build check. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。