#6490 [Engine] Load pack weights directly into shared Metal storage

open · draft · @Maryarty · 0 コメント · GitHub で見る

Server & APIAMD / HIPNVIDIA / CUDAModels & quants

本文

## Problem and result

The Metal pack loader stages weights in pinned buffers before copying them into a shared Metal arena. A checked CPU view can receive the same bytes directly. This removes the 16 MiB output staging allocation during pack loading and avoids the copies into the arena.

## Related discussion

The resident dense-buffer lifetime work is tracked in #6440.

## Approach

`STRATA_METAL_PACK_DIRECT=1` enables the path when the complete destination range has a usable Metal CPU mapping. Raw planes are read into that mapping. BF16 and FP16 narrowing and FP16 scale-plane widening write their output there using the existing conversions and one 8 MiB input staging buffer. The staged path also handles missing mappings.

## Validation

Base `3142b66`, submitted head `dda3ffa`. Apple M1 Max, 32 GPU cores, 64 GiB, macOS 26.6.2.

**Passed**

- Release build with `cmake --build engine/build-metal --target strata pack_direct_test runtime_hello moe_tile_grid_test -j 6`.
- `ctest --test-dir engine/build-metal --output-on-failure` passed all four tests. `pack_direct` compares the bytes in the full and compacted arenas and checks raw data, BF16, FP16 and widened scales.
- Four fresh `strata --serve` processes loaded the same Qwen3.8-Flash-Next GSQ-RCO Q2_0 with PLE, streaming arena 12 GB, context 512, spec 2, MTP and suffix drafting disabled. Each processed the same 16 input IDs and returned 128 greedy IDs. All first-logit checksums and output IDs matched. Each arm recorded zero new system Swapouts.

| Order | Pack mode | Reported pack load | Time to READY |
| --- | --- | ---: | ---: |
| 1 | Staged | 1.9 s | 34.60 s |
| 2 | Direct | 0.2 s | 6.15 s |
| 3 | Direct | 0.2 s | 4.99 s |
| 4 | Staged | 0.7 s | 6.07 s |

**Failed**

- None observed in this series.

**Not run**

- CUDA, HIP and SYCL builds were unavailable on this Mac. This stays in draft for backend build coverage.

## Limits

The pack timing includes the observed file-cache and process-start variability. Time to READY spans several phases and varied substantially. The memory saving is limited to the temporary pack-loading output buffer. This run used one model family on one Metal device.

関連リンク

インストール・モデル・リリースへのサイト内リンク。