Pull requests / #281

#281 numerics: saturate the prompt path's FP16 SwiGLU products; log1pf in the unfused GDN gate

closed · @sergqwer · 0 comentários · No GitHub

NVIDIA / CUDAWindows

Descrição

## Summary

Two numerical edge cases found in a read-only audit of the prompt path:

- **FP16 overflow in SwiGLU.** The products that feed the prompt path's FP16 down projection are now clamped to
  FP16's range (±65504) before rounding. A token with a massive activation otherwise becomes `inf` there, and `NaN`
  after the GEMM. Decode's q8_1 path (range ~8e6) stays finite for the same token.
- **Softplus underflow.** The unfused GDN gate used `logf(1 + expf(v))`, which returns 0 for v below ~-16 (1 + e^v
  rounds to 1). The fused kernels, `gdn.cu` and `verify_kernels.cu` already use `log1pf(expf(v))`; now the unfused
  path does too.

## Measured

Measured on current main (0.1.29, d6708a4) with ISTA-DASLab's IQ2_XS, RTX 5090 32 GB, Ryzen 9 9950X3D, 128 GB DDR5, Windows 11. KL is first-token KL divergence (`STRATA_DUMP_FIRST_LOGITS`, #276); two runs of main itself differ by 0 on 5 of 6 short prompts and by 0.00013 on the sixth. On ordinary prompts nothing moves: first-token KL to main 0, 0.00013 and 0 on three prompts
(1K, 2K and 4K tokens), which is the run-to-run noise itself.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

No site

Links install, modelos, releases.