Pull requests / #1732

#1732 prefill: the gfx906 build reached the CUDA branch of arm_cpu_share

open · @phoenixclyde · 0 comentarios · En GitHub

AMD / HIPNVIDIA / CUDA

Descripción

Prefill::arm_cpu_share() asks "is this AMD?" with
#if !defined(STRATA_USE_HIP), but the gfx906 compat build (PR #638)
compiles as HIP *without* STRATA_USE_HIP -- see src/core/expert_cache.cpp
:5-6, which guards the same question with
#if defined(STRATA_USE_HIP) || defined(STRATA_HIP_GFX906).

On gfx906 the CUDA branch is therefore live: the CPU share is armed by
default for chunks below 1,024 tokens and prints its banner, against this
file's own note at :161 ("the layer split, batch slots and HIP stay off").
Seven other sites already carry the || defined(STRATA_HIP_GFX906) form.

Measured on one MI50 32 GB (gfx906), CLI generate, a 4-token prompt, one
binary with one set of weights and one config, only the environment
differing, power saving off:

  default  experts streamed 309 (295 by DMA, host 174.0 ms), resident 607,
           on the CPU 197 (share 0.90)          prefill 928.3 ms
  =0       experts streamed 506 (492 by DMA, host 177.3 ms), resident 606,
           on the CPU 0 (share 0.00)            prefill 929.5 ms

Output token ids are identical (271 248068 198 760) and the 0.13%
difference is inside this machine's run-to-run spread (1.2-1.7%), so on
this card the share engages and buys nothing.  #1595 and #1729 report the
default costing on CUDA; this is the mechanism that turns it on for AMD.

CUDA and SYCL do not define STRATA_HIP_GFX906, so the added check is a
no-op there and nothing on those backends changes.

Hardware the numbers come from: Xeon E5-2696 v4 (Broadwell, 22C/44T, no
AVX-512), one Radeon Instinct MI50 32 GB (gfx906, PCI 1002:66a1, VBIOS
113-D1631711-100, 34,342,961,152 B of VRAM), PCIe 3.0 x16 effective (the
root port runs 8 GT/s x16 and the bridge above the card is downgraded;
the card's own hop negotiates 16 GT/s, about 15.75 GB/s end to end),
128 GB RAM, Ubuntu, kernel 7.0.0-34-generic, ROCm 7.14 in a container,
built with -DSTRATA_HIP_GFX906=ON and -DCMAKE_HIP_ARCHITECTURES=gfx906.

En el sitio

Enlaces a install, modelos, releases.