#6374 [Engine] Scope: reuse normalized BF16 hyper-connection inputs on NVIDIA GB10

open · @InB4DevOps · 2 comments · View on GitHub

Multi-GPUNVIDIA / CUDAModels & quantsLinux

Description

On one DGX Spark (GB10, SM 12.1), the BF16 hyper-connection read computes the normalized residual during the down projection and reconstructs it again in the up projection. I would like to propose using the existing normalized scratch in both stages for the staged 1–6-token verify path on this device. The narrow path also keeps down-dot partials in registers and groups eight output columns in the up projection. Other formats and devices would keep their existing paths.

Four alternating warm-server pairs on stock `58ee32277`, Qwen3.8-Flash-Next IQ3_S, 128K context and default DFlash2, measured +2.91%, +3.12%, +2.49%, +2.85% primary decode in the prototype (+2.84% geometric mean). This is below the original 3% target. Short-prompt prefill was 1.97% slower on average. The prototype matched the 256 output IDs, 178/277 accepted/offered primary drafts, and verifier-logit dump. These are not measurements against current main.

Is an automatic GB10-only path with the existing plain/split fallback acceptable in scope? Draft PR: #6375. Scope approval is pending.

### Hardware and operating system

- **GPU / VRAM:** NVIDIA GB10 (DGX Spark, compute capability 12.1). The GPU shares the system’s 128 GB unified memory; it has no separate dedicated VRAM. `nvidia-smi` reports total GPU memory as `N/A`.
- **System RAM:** 128 GB unified memory, shared with the GPU (about 121.7 GiB visible to Linux; `/proc/meminfo` reports 127,598,700 kB).
- **OS:** Ubuntu 24.04.5 LTS, Linux `7.0.0-1024-nvidia`, aarch64. NVIDIA driver `580.178.04`; CUDA compiler `13.0.88`.

Related on this site

Quick links to related pages.