Pull requests / #3537

#3537 [Other] setup (Mac): Swift 1.5 IQ3_S, opt-in with `--family swift`

open · draft · @oceanio · 0 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIModels & quantsDocumentation

Beschreibung

## Problem and result
On a Mac, `./setup.sh` sets up the original Qwen3.8-Flash-Next and Unsloth's UD files only; docs/MAC.md says "The Swift
and Coder releases are not set up on Mac yet". Swift 1.5 itself runs on the Metal engine, but setup cannot prepare it:
its two shards cut through a layer (IQ3_S: layer 14's gate in shard 1, its up and down in shard 2), shard 1 also holds
the n-gram table, and the Metal engine maps one file.

With this PR, `./setup.sh --family swift` downloads Swift 1.5 IQ3_S, repacks its shards into one GGUF with the existing
`metal/tools/metal_repack.py` (as setup already does for Unsloth's files), builds the pack and writes a config. Without
`--family swift` nothing changes: the size menu, the original model and Unsloth's files.

## Related discussion
#2898 (feature request). No maintainer reply on scope yet: opened as a draft so the change can be read beside the request.

## Approach
- Whether a model is repacked now follows `repacks()` (shards that cut through layers: Unsloth's 4-6, Swift's 2)
  instead of the shard count. The n-gram table stays in Swift's shard 1.
- `--family swift` takes IQ3_S only, the size measured on a Mac; another size stops setup with the command to run.
  Images are off for Swift (the image encoder is checked with the original model only), with a note when `--vision`
  asks for them (yes, gpu or cpu); the original model's default stays images on.
- Swift's pack size lives in `strata_control/mac_fit.py` (`MAC_SWIFT`, `mac_model()`, and an optional `family`
  argument on `model_need_gb` / `streams` / `model_note`). The existing callers (`setup.py`'s Control mode,
  `tools/strata_mcp.py`, `strata launch`) pass `(model, ram)` and see no change.
- No engine change: the pack, the MTP draft layer and the engine are the same. `iq_pack` writes Swift's 96 F32 routers
  as BF16 (all exact), and the draft layer is the original model's, as on a PC.
- docs/MAC.md: the `--family swift` option, what is measured, and the Families line.

## Validation
Environment: M5 Max 128 GB, macOS 26.7.1, Command Line Tools 27.0 (Apple clang 21).

**Passed:**
- `python3 tools/test_setup_mac.py` on d34812e4 (= `main` 9de04cd4 + this PR): 30 tests OK, 6 of them new (the
  original's and Unsloth's file names and repack decision unchanged; Swift's file names, its repack, its engine
  arguments; IQ3_S with images off; an unmeasured size stopping; the choices without `--family swift`).
- `python3 -m unittest tools.test_strata_mcp.MacCatalog tools.test_strata_mcp.Helpers` (the tests that import
  `mac_fit`): OK on this PR; `python3 tools/test_setup_menu.py`: OK.
- Swift 1.5 IQ3_S on the Metal engine (SC117's abliterated build, the same two-file layout as UkisAI's release),
  repacked file 54.8 GB: decode 93.1-110.4 tok/s on 4 prompts (Qwen IQ3_S 94.3-109.2 on the same Mac), prompt
  1,636-1,731 tok/s, 6.8K-token needles 3 of 3, 3 reasoning questions right with thinking on. Measured on the 0.1.45
  engine with the Mac build fix that 0.2.0 carries.
- The engine built from this branch (= `main` 9de04cd4, no engine change; `cmake --build build-metal --target strata`,
  0 errors) with the repacked Swift file, through this branch's `serve/server.py` at `--max-context 262144`: answers a
  short question ("Paris"), and `tools/needle_bench.py --lengths 2k,16k,32k --depths 10,50,90` finds 9 of 9
  (1,644 / 15,070 / 31,663 prompt tokens). The same check passed on 0.2.0-beta (`main` e88ee39d) before the rebase.

**Failed:** none observed.

**Not run:**
- `tools/test_strata_mcp.py` in full: its fake server uses port 8080, which a running Strata held on this Mac.
- A fresh end-to-end `./setup.sh --family swift` on 0.2.1.4 (55 GB download and repack). It ran end to end
  (`--gguf-dir`) before the rebase, on 0.1.45 with the Mac build fix.
- UkisAI's own files (SC117's build has the same layout), other Macs, streaming on a smaller Mac, quality beyond 32K
  tokens, the KL harness, the Coder release.

## Limits
- IQ3_S only; other Swift sizes stop setup with a message until they are measured on a Mac.
- Images off for Swift on a Mac.
- Opt-in only: the size menu, the app and `strata launch` do not offer Swift.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.