#6088 [Engine] Nvidia v100 performance slow down -12% with "own kernels"
open · @rolsch · 4 comentários · No GitHub
BenchmarksAMD / HIPNVIDIA / CUDA
Descrição
Hardware: 6× Tesla V100 16 GB (SM70) Vor 0.2.14: Engine 0.1.58 (Strata 0.2.12, alter llama.cpp/MMQ-Promptpfad) Nach 0.2.14: Engine 0.1.60 (Strata 0.2.14, neue Own-Kernel-Pipeline) V100 Comparison (Tesla V100 16 GB, SM70) Before the Own Kernel transition: Strata 0.2.12 Engine 0.1.58 Legacy MMQ/llama.cpp prompt path Prompt throughput: approximately 862 tokens/s Total runtime: approximately 122-125 seconds After the Own Kernel transition: Strata 0.2.14 Engine 0.1.60 New Strata Own Kernels Prompt throughput: approximately 769 tokens/s Total runtime: approximately 137-139 seconds Difference: Prompt throughput: about 11% slower Total runtime: about 12% longer Summary: On the Tesla V100 (Volta, SM70), published community benchmarks have not shown an advantage from the new Own Kernel pipeline so far. While newer GPUs such as the RTX 4090, RTX 5090, and AMD Strix Halo appear to benefit from the changes introduced in Strata 0.2.14, the V100 performs somewhat better with the legacy MMQ/llama.cpp path. For this reason, building Strata with -DSTRATA_MMQ=ON may still be worth considering for Volta-based systems. This issue should be taken into account during Strata's optimization run. On NVIDIA V100 (SM70) hardware, the optimizer should consider switching back to the MMQ/llama.cpp path when the new own kernels result in lower prompt throughput or higher end-to-end latency. - or, optimize the own kernels for v100 hardware (feature switch)
No site
Links install, modelos, releases.