#6479 [Engine] CPU/GPU SwiGLU discrepancy from exponential evaluation on NVIDIA Ampere
open · @KiralyCraft · 6 コメント · GitHub で見る
本文
**I've investigated this with AI, the report below is written by AI but audited by myself** I’m investigating numerical differences when Strata evaluates the same expert on the CPU and GPU. In a standalone replay on an RTX 3050 Laptop GPU (`sm_86`), I fed identical FP32 gate and up values into both SwiGLU paths. The first difference was in the exponential: CPU `std::exp(-gate)` returned `0x409e9198`, while CUDA `__expf(-gate)` returned `0x409e9199`. That one-ULP difference survived the rest of SwiGLU: the hidden value was `0x3e521115` on CPU and `0x3e521113` on CUDA. Feeding both hidden vectors through the same quantizer produced different codes, `103` and `102`, at that lane. I suggest checking this on your hardware with a stock Strata build. Feed the CPU and CUDA SwiGLU paths the same saved FP32 gate and up values, then compare the raw bits after the exponential, denominator, division, and final multiplication. If the hidden values differ, pass both through the same quantizer to see whether the difference survives. Comparing the stages separately helps distinguish an exponential difference from a quantizer rounding difference. This is a kernel-level result on one Ampere GPU. The question is whether this reproduces in stock builds and whether moving an expert between CPU and GPU is expected to preserve its numerical output. This paper solves this issue by "emulating" CUDA's behavior on CPU: https://arxiv.org/abs/2611.13879 It also has an implementation here: https://github.com/hzX40000000/SFU-Sim
関連リンク
インストール・モデル・リリースへのサイト内リンク。