#6540 [Engine] Hy4-Flash: SQ-96 starts on 128 GB Macs, --info no longer crashes, session reuse on by default, reasoning_effort max in configs

open · @mnott · 0 comments · View on GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDA

Description

## Problem and result

Four Hy4-Flash fixes found running Hy4 on an Apple M5 Max 128 GB. **Stacked on #6539** (the first three commits of this branch are that PR; review the last four). The two engine fixes and the serve fix do not depend on it and can be split out if preferred; the clamp below accounts for the conversation slots of #6539.

1. **SQ-96 does not start on a 128 GB Mac.** The SQ-96 file (89.6 GB) exceeds Metal's maxBufferLength (86.6 GB here), so the runner takes the cold-expert path. Its LRU expert cache was sized `workingSet - allocated - 4 GB` (99.6 GB here) and clamped only to the total expert size, so the single allocation failed: `mtl: alloc 99626901504 failed`, exit. Only `STRATA_METAL_EXP_CACHE_GB` got it running. Now the budget is clamped to maxBufferLength minus 1 GB and reduced by the KV sets the conversation slots will allocate. After: SQ-96 starts without the variable: `[cold-experts] LRU cache 85.51 GB (17485 slots x 4.89 MB)`, `metal allocated 97.48 GB`, answers correctly.
2. **`strata-hy4 --info` crashed** (SIGSEGV, rc 139) when run without a model, the way serve probes the engine. Now it prints `arch=hy4flash drafter=dfly2 images=1 backend=metal` and exits 0.
3. **Session reuse on by default.** setup wrote Hy4 run configs without `--session-reuse`, and serve passed none, so prompt reuse (and with #6539 the slots and SSD tier) was off unless the user knew the flag. setup now writes `--session-reuse 1`, and serve adds it for a Hy4 config that does not set it; an explicit `0` still turns it off.
4. **`reasoning_effort: "max"` in a Hy4 run config** was refused at start (`config reasoning_effort: none, low, medium or high`), although the Hy4 template supports `no_think | high | max`. Hy4 configs now accept `max`; other families keep today's validation.

## Related discussion

none (summary issue with the findings linked from #6539).

## Approach

- Clamp in the `if (g.cold_experts)` block of `hy4_metal_main.cpp`; the env override stays and is clamped too.
- `--info` no longer touches model state before printing the build's capabilities.
- Reuse default in `terminal/setup.py` (Hy4 config writer) and `serve/hy4_kind.py` (`engine_args`), with the explicit value winning.
- `max`: `clean_shared_defaults` takes a Hy4 flag (`server_parts/http_api.py`, `server.py` passes `is_hy4(cfg)`).

## Validation

**Passed:**
- Build: `engine/src/hy4flash/metal/build_mac.sh run` on this branch (v0.2.18 base), macOS 27.3, Apple clang, M5 Max 128 GB; `strata-hy4-metal --info` → INFO line, rc 0.
- Live: SQ-96 run config without `env`, this binary, `--conversation-slots 2`: loads in 10.7 s (cache line above), serve added `--session-reuse 1` by itself, a coding prompt answered correctly.
- Live: a Hy4 config with `"reasoning_effort": "max"` starts ("thinking level: max") and a full agent task ran at max.
- `python3 -m pytest serve/test_server.py serve/test_hy4_dialect.py serve/test_effort_paths.py serve/test_hy4_serve_e2e.py -q`: 498 passed, 734 subtests.

**Failed:**
- `tools/test_setup_hy4.py::MacHy4Flash::test_mac_install_plan_defaults_to_128k_and_unified_fit` fails here, and also on a clean v0.2.18 checkout (not caused by this PR).

**Not run:**
- CUDA/HIP/SYCL engines (fix 1 and 2 are in the Metal runner; the reuse default and `max` are in serve/setup and apply to all).

## Limits

- At `max`, Hy4 can keep reasoning in the answer channel once the thinking budget is used (13-18k-token answers seen); a `reasoning_budget_tokens` does not bound that part.
- In cold-expert mode (SQ-96, SQ-128 on 128 GB) prompts are much slower than on SQ-64, which maps whole (~260 vs ~1,070 tok/s here); the setup recommendation for 128 GB Macs may want to account for maxBufferLength.

Related on this site

Quick links to related pages.