问题 / #2541

#2541 [Feature]: Skip low-rank missed experts on Metal when streaming from the SSD

open · @qChauncey · 0 评论 · 去 GitHub 看

BenchmarksNVIDIA / CUDA

说明

Get faster decode on a Mac that streams experts from the SSD. On my Mac Studio (M4 Max, 36 GB, 512 GB SSD, UD-Q4_K_XL, 16 GB wired arena), decode is about 11 tok/s. The SSD reads 2.3–3.5 GB/s during decode, which looks like its limit for this access pattern. The arena holds 5,022 of 24,576 experts.

On CUDA, STRATA_ROUTE_TAIL_SKIP targets this same cost, but I can’t find it in metal/. Could it be ported to the Metal streaming path? I have not measured what it would gain here.

本站相关内容

相关页面的快捷入口。