Issues / #2898

#2898 [Feature]: Swift 1.5 on a Mac (opt-in --family swift, IQ3_S)

open · @oceanio · 0 comentários · No GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descrição

### What do you want to do?

Run Swift 1.5 on a Mac. docs/MAC.md says the Swift release is "not set up on Mac yet", and `metal/setup_mac.py`
only takes `--family qwen|unsloth`.

**What blocks it.** Swift's two shards cut through a layer: in IQ3_S, layer 14's gate is in shard 1, and its up and
down are in shard 2. Its shard 1 also holds the n-gram table. The Metal engine maps one file (`window.cpp` wraps
shard 1), so it cannot read Swift as it is.

**What already works.**
- `metal/tools/metal_repack.py` (written for Unsloth's 4-6 shards) works on Swift's 2 shards as they are.
- `tools/iq_pack.py` already handles Swift's layout. It writes the 96 F32 routers as BF16, and all 96 are exact.
- The Metal gemv has IQ4_NL and IQ4_XS, which Swift uses for `attn_output`, `ssm_out` and `ffn_down_shexp`.
- The router read takes F32.
- The engine needs no change.

**What I propose (opt-in).** `./setup.sh --family swift` sets up Swift 1.5 IQ3_S: setup repacks it like Unsloth's
files, and leaves images off. Without `--family swift` nothing changes, including the size menu, the original model
and Unsloth's files. Only IQ3_S is offered, because it is the only size I could measure. I have this on a branch:
https://github.com/oceanio/Strata/tree/feat/mac-swift
- One commit on `metal/setup_mac.py`, `tools/test_setup_mac.py` (26 tests, 6 new) and `docs/MAC.md`.
- It sits on #2895: main does not build on a Mac without it (#2860).

**Measured** on an M5 Max (128 GB, macOS 26.7.1) with SC117's abliterated Swift 1.5 IQ3_S
(`SC117/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF`, the same file layout as UkisAI's release). The model
was resident, with `--spec 2` and MTP. Engine 0.1.45 with #2895, installed by the branch's setup end to end
(`--gguf-dir`).

| | Swift 1.5 IQ3_S | Qwen3.8-Flash-Next IQ3_S (SC117), same Mac |
|---|---|---|
| Decode, 4 prompts up to 200 tokens, greedy | 93.1 / 97.2 / 110.4 / 104.6 tok/s | 94.3 / 95.1 / 109.2 / 108.5 tok/s |
| 6.8K-token needle, 3 documents | 3 of 3 | 3 of 3 |
| Prompt, 6.8K tokens (warm) | 1,636-1,731 tok/s | 1,651-1,767 tok/s |
| 3 short reasoning questions, thinking on | 3 of 3 right | 3 of 3 right |

**Not tested.**
- UkisAI's own files: I only have SC117's.
- A comparison with the CUDA or HIP engine on the same Swift file. I have no NVIDIA or AMD card.
- The KL harness.
- Contexts over 8K, streaming on a Mac under ~96 GB, a long server run, and sizes other than IQ3_S.

**Questions.**
1. Would you take this as an opt-in setup change?
2. Could someone with a CUDA or HIP card give reference outputs for Swift IQ3_S, for example greedy text on a few
   prompts? I would then compare the Metal engine against them, the way docs/MAC.md does for the original model.

I would open the PR once #2895 is in, so that it holds only this commit.

No site

Links install, modelos, releases.