#6490 [Engine] Load pack weights directly into shared Metal storage
open · draft · @Maryarty · 0 Kommentare · Auf GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
## Problem and result The Metal pack loader stages weights in pinned buffers before copying them into a shared Metal arena. A checked CPU view can receive the same bytes directly. This removes the 16 MiB output staging allocation during pack loading and avoids the copies into the arena. ## Related discussion The resident dense-buffer lifetime work is tracked in #6440. ## Approach `STRATA_METAL_PACK_DIRECT=1` enables the path when the complete destination range has a usable Metal CPU mapping. Raw planes are read into that mapping. BF16 and FP16 narrowing and FP16 scale-plane widening write their output there using the existing conversions and one 8 MiB input staging buffer. The staged path also handles missing mappings. ## Validation Base `3142b66`, submitted head `dda3ffa`. Apple M1 Max, 32 GPU cores, 64 GiB, macOS 26.6.2. **Passed** - Release build with `cmake --build engine/build-metal --target strata pack_direct_test runtime_hello moe_tile_grid_test -j 6`. - `ctest --test-dir engine/build-metal --output-on-failure` passed all four tests. `pack_direct` compares the bytes in the full and compacted arenas and checks raw data, BF16, FP16 and widened scales. - Four fresh `strata --serve` processes loaded the same Qwen3.8-Flash-Next GSQ-RCO Q2_0 with PLE, streaming arena 12 GB, context 512, spec 2, MTP and suffix drafting disabled. Each processed the same 16 input IDs and returned 128 greedy IDs. All first-logit checksums and output IDs matched. Each arm recorded zero new system Swapouts. | Order | Pack mode | Reported pack load | Time to READY | | --- | --- | ---: | ---: | | 1 | Staged | 1.9 s | 34.60 s | | 2 | Direct | 0.2 s | 6.15 s | | 3 | Direct | 0.2 s | 4.99 s | | 4 | Staged | 0.7 s | 6.07 s | **Failed** - None observed in this series. **Not run** - CUDA, HIP and SYCL builds were unavailable on this Mac. This stays in draft for backend build coverage. ## Limits The pack timing includes the observed file-cache and process-start variability. Time to READY spans several phases and varied substantially. The memory saving is limited to the temporary pack-loading output buffer. This run used one model family on one Metal device.
Mehr auf der Site
Links zu Install, Modellen, Releases.