Pull requests / #281
#281 numerics: saturate the prompt path's FP16 SwiGLU products; log1pf in the unfused GDN gate
closed · @sergqwer · 0 commentaires · Sur GitHub
Description
## Summary Two numerical edge cases found in a read-only audit of the prompt path: - **FP16 overflow in SwiGLU.** The products that feed the prompt path's FP16 down projection are now clamped to FP16's range (±65504) before rounding. A token with a massive activation otherwise becomes `inf` there, and `NaN` after the GEMM. Decode's q8_1 path (range ~8e6) stays finite for the same token. - **Softplus underflow.** The unfused GDN gate used `logf(1 + expf(v))`, which returns 0 for v below ~-16 (1 + e^v rounds to 1). The fused kernels, `gdn.cu` and `verify_kernels.cu` already use `log1pf(expf(v))`; now the unfused path does too. ## Measured Measured on current main (0.1.29, d6708a4) with ISTA-DASLab's IQ2_XS, RTX 5090 32 GB, Ryzen 9 9950X3D, 128 GB DDR5, Windows 11. KL is first-token KL divergence (`STRATA_DUMP_FIRST_LOGITS`, #276); two runs of main itself differ by 0 on 5 of 6 short prompts and by 0.00013 on the sixth. On ordinary prompts nothing moves: first-token KL to main 0, 0.00013 and 0 on three prompts (1K, 2K and 4K tokens), which is the run-to-run noise itself. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
Sur le site
Liens install, modèles, releases.