#3271 [Other] gfx906 (2× Radeon Pro VII): install report, 200K k8v4 + CPU vision, and a 50% CPU question

Description

### Area

[Other]  (anything else, including setup and installation)

### Your question

Two minor questions; everything else works:

1. **CPU use:** `top` peaks at ~50% (18 of 36 logical CPUs) on this 18C/36T Xeon. The engine log says *17 pool workers + the host thread*; `docs/AMD_HIP.md` and `docs/DETAILS.md:557` suggest one worker per physical core is deliberate and all-logical was slower on gfx1030. Is that the expected behavior on Haswell-EP too?
2. **Setup path:** the docs say setup does not build gfx906, but on v0.1.45 setup stops at its first GPU check, so the model download and pack preparation are also unreachable without patching `setup.py`. Is a **download-only** route (or clearer by-hand download instructions) planned? That was the only real install dead end I hit.

The report below is a successful end-to-end run on v0.1.45 (`477ac13c`), 2× Radeon Pro VII 16 GB (gfx906), 64 GB RAM, CachyOS kernel 7.2.3: Qwen3.8-Flash-Next IQ3_S, 200K context, `--kv k8v4`, pictures on (CPU encoder). It is also posted as a comment on #2883 (the gfx906 validation thread); #3172 tracks the gfx906 build blockers I hit. I am not asking for a new feature; I am sharing what worked and where the friction is.

### What you tried

#### Environment

| | |
|---|---|
| GPUs | 2× AMD Radeon Pro VII 16 GB, gfx906 (Vega 20, `1002:66a1`), 190 W power limit, PCIe 3.0 (engine probe 13.8 GB/s host→device) |
| CPU / RAM | Intel Xeon E5-2696 v3, 18C/36T, AVX2; 64 GB DDR4-2133 ECC at 1866 MT/s (setup reports 63 GB) |
| OS | CachyOS (Arch-based), kernel 7.2.3-1-cachyos, GCC 16.2.1 |
| ROCm | system ROCm 7.2.4 at `/opt/rocm`, HIP 7.2.53211, AMD clang 22.0.0; no container, no `HSA_OVERRIDE_GFX_VERSION`. The distro package ships gfx906 rocBLAS kernels (`Kernels.so-000-gfx906-xnack-.hsaco`); the community image was not needed. |
| Strata | source v0.1.45, commit `477ac13c`, engine `0.1.45`; llama.cpp `3cf03257f` |
| Model | Qwen3.8-Flash-Next GSQ-RCO IQ3_S, two shards 54.82 + 28.80 GB; vision `mmproj-Qwen3.8-Flash-Next-BF16.gguf` on CPU |
| Config | `--max-context 204800 --kv k8v4 --vision --gpus 0,1 --layer-split auto --spec 4 --spec-min-p 0.5 --prefill auto --expert-cache auto --vram-reserve-mib 700 --vram-reserve-later-mib 1006` |
| MTP | draft layer reused from an earlier install under `Strata-data/mtp/rt`; `tools/mtp_fetch.py verify` rc=0 |

Setup command: `./setup.sh --yes --backend hip --gpus 0,1 --family qwen --model IQ3_S --vision cpu --no-start`, then the 200K / `k8v4` choices.

Note: setup's step-2 summary prints `KV cache: 4-bit (Hadamard-rotated)` for any non-int8 choice (`setup.py:6581`); the written config is `--kv k8v4` (K 8-bit, V 4-bit). Wording only.

#### What worked

- **Build:** the documented `-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906` path, with the compat layer, built the engine and the CPU vision encoder. Build time ~15 min at `-j18`. The 4-file diff is in the gist below; the CMake + cublas hunks are the same set posted in #2459, and #3172 tracks the same blockers.
- **KV parity on the cards:** `kv_hybrid_parity` **ALL PASS** (k8v4 appends/gathers, 0 bad of 786,432/24,576; mode-3 batched attention 7.82e-08 vs the dequant reference), `kv_stream_parity` k8v4 ring 0 failures, `kv_window_parity` 0 bad/0 stale, `kv_steps_parity` 120/120, `kv_q8_parity` OK.
- **End-to-end:** setup (patched) → engine → download → pack → start script; server `/health` ready; text `17*23?` → `391`; vision (64×64 red PNG) → `Red`; images on. The docs' gfx906 section lists images (`--vision`) as not done — this run verifies the CPU encoder path on Radeon Pro VII.
- **Start:** ~75–90 s to ready; 46.84 GiB of experts into RAM at 1.51 GiB/s; expert cache 5,724 slots (10.30 GiB) and, on CUDA1, 4,180 slots (8.42 GiB); 586 MiB / 768 MiB free after load.
- **200K config:** `k8v4` at 204,800 costs ~2.2 GB and stays in VRAM. KV streaming is off: it needs ~2.2 GB beside the ~62 GB of experts on a 63 GB machine. No KV-related errors.
- **Long sessions:** several chats with 23K–35K-token prompts and 22K–42K-token outputs; decode held 43.8–51.6 tok/s (table below). The requests completed; the only warnings were the VRAM notes.
- **Downloads / MTP:** 86 `download interrupted` retry lines across 13 setup attempts (mostly DNS, 12 read timeouts) on a flaky link; every resume continued from `.part`, zero corruption. MTP verify passed.

#### What did not work / known issues

**Known / already tracked**

1. **`setup.py` refuses gfx906 at step 1** — documented as "Setup does not build it yet", but the failure also makes the ~84 GB download and pack preparation unreachable through setup. Exact first-run output (before the local patch):

   ```
     Your AMD GPUs:
       GPU 0: AMD Radeon (gfx906), 16 GB VRAM - cannot be used
     [!] AMD GPU 0 (AMD Radeon (gfx906)) cannot be used: not supported - [...]
         this is gfx906
     Setup stopped. Fix the item above and run it again - everything already done is kept and skipped.
   ```

   My local `setup.py` change adds gfx906 to `AMD_ARCHS`/`AMD_NAMES` and passes the gfx906 flags (including `-DSTRATA_ENABLE_HIP=OFF`). I am **not** proposing it as the default (the docs say manual build); a **download-only** path, or clearer by-hand download instructions, would remove the dead end. This is the main friction I hit.

2. **Build blockers on 0.1.45** (reproduced locally; exact messages in `build-errors.txt` in the gist):
   - missing `cublasSgemm`, `cublasSgemmStridedBatched`, `cublasGemmStridedBatchedEx` aliases in `platform/hip_compat/cublas_v2.h` (qwen35 `q36_engine.cu`);
   - `strata_mmq_w4a4` compiled without `GGML_USE_HIP` → `cuda.h` not found;
   - `f16kq::*` undefined at link — missing `src/prefill/moe_f16_kq.cu` in `strata_mmq` under `STRATA_HIP_GFX906`;
   - GCC 16 `<format>` vs HIP's `__noinline__` macro (fix: `#undef __noinline__` in the gfx906 compat header);
   - stale `build-hip` cache with `STRATA_ENABLE_HIP=ON` + `STRATA_HIP_GFX906=ON` → "enable only one"; pass `-DSTRATA_ENABLE_HIP=OFF` or use a fresh build directory.

   These are in **#3172** (open; confirmed on 0.2.0-beta) and were reported in **#2459** (closed by the tracker reset). I am adding the end-to-end result, not a new build bug.

**Rough edges found here**

3. **VRAM is tight at request start.** The first start (only `--vram-reserve-mib 700`) logged `CUDA1: 462 MiB ... LOW`; adding `--vram-reserve-later-mib 1006` removed it (768 MiB free after load). One later request still logged:

   ```
   WARNING: only 493 MiB of VRAM free at the start of this request (below 512 MiB) ... Add --vram-reserve-mib 768 ...
   ```

   The following long requests still ran at 44–52 tok/s, but if the driver starts paging, `--vram-reserve-mib 768` (or 1024) is the engine's own suggestion. The desktop here is on CUDA1; `--vram-reserve-later-mib 1006` is what fixed the CUDA1 window-capture warning.

4. **`qsa_prompt_attn mode 3` refuses on HIP** and the older kernel runs; output is correct, but if mode 3 is expected to work on gfx906 this is masking it. (Observed in the parity tests; not a correctness failure.)

5. **No hipBLASLt tuning table for gfx906** (hipBLASLt 100202): setup says the prompt's dense GEMMs use plain hipBLAS. Expected for this arch, but it is a prompt-speed opportunity.

6. **Network:** the only reason setup needed 13 attempts; not a Strata issue, but the log is full of it.

#### CPU usage: ~50% (18 of 36 threads)

`top` peaks around 50% on this 18C/36T Xeon. The engine log says:

```
strata generate: 17 pool workers on logical processors 1,...,17, the host thread on 0 (--host-core first)
strata aux cpus: ... CPUs 18 (off at start; a request's aux_cpus=1 turns it on)
```

So it uses one logical CPU per physical core (17 workers + host = 18 threads) and leaves the 18 SMT siblings alone. `docs/AMD_HIP.md` (gfx1030 report) says the default is one worker per physical core and that using all logical CPUs was slower there (27 vs 38–42 tok/s); `docs/DETAILS.md:557` documents `--pool-workers`. I think this is expected, not a bug. If a sweep on Haswell-EP would be useful, I can run `--pool-workers 8/17/24/35` with a fixed prompt, but I have not done it.

#### Not tested

- **0.2.0-beta** (`3d1fb480`): not run; #3172 says the gfx906 build still fails there. This report is v0.1.45 only.
- Full 200K prompt / needle tests; KV streaming on; concurrent requests; a 12 h soak; a formal 3-run benchmark or `tools/strata_bench.py`; vision accuracy/speed beyond one image; CUDA/SYCL builds; Windows; a fresh MTP download.
- I did not run the CUDA build. My CMake changes are inside `STRATA_HIP_GFX906`, and the cublas/`__noinline__` changes are in the gfx906 compat headers, but I cannot claim the other backends from this machine.

#### Logs

Sanitized files in one secret gist (link-only, not indexed): `setup-evidence.log` (curated; the full setup log is 1,193 lines), `engine-session-evidence.log`, `build-errors.txt`, `strata-gfx906-fixes.patch`, `config-iq3_s-sanitized.json` — https://gist.github.com/Marczellito/9e9324702246fe9f1acfb0306e868c9e. Local paths are shortened to `~`; no API keys, passwords, emails, or hostnames. Full logs on request.

### Results (optional)

From `strata-iq3_s.log` after the 1006 restart. These are engine-reported averages from one continuous session, **not** the 3-run benchmark protocol. "Prompt read" is the newly read part; the reused prefix is excluded, so the prompt speed is the speed of that read, not of the whole context.

| prompt tokens (reused + new) | prompt speed | generated | decode | drafts accepted | cache hit | end context |
|---|---|---|---|---|---|---|
| 23,133 (81 + 23,052) | 590.3 tok/s | 41,406 | 51.6 tok/s | 25,704 / 33,271 | 97.4% | ~64.5K |
| 31,231 (23,128 + 8,103) | 403.8 tok/s | 33,051 | 43.8 tok/s | 17,267 / 27,890 | 97.2% | ~64.3K |
| 32,910 (31,226 + 1,684) | 295.7 tok/s | 42,604 | 50.3 tok/s | 27,297 / 38,890 | 96.9% | ~75.5K |
| 34,916 (32,905 + 2,011) | 312.2 tok/s | 22,758 | 50.4 tok/s | 14,153 / 17,521 | 94.4% | ~57.7K |
| 500 (56 + 444) | 157.5 tok/s | 2,829 | 48.2 tok/s | 1,638 / 2,453 | 94.5% | — |

- First request after start: 50 tokens at **12.9 tok/s** (graph capture), then 50.3 tok/s on the next one. Do not judge from the first request.
- Tiny prompts (61–86 tokens) read at 46–76 tok/s; short prompts are fixed-overhead bound, as the docs say.
- The 200K limit was **configured**, but the largest verified request ended at ~75.5K total tokens; I did not fill 200K. Anyone reading "200K" as tested should know this.
- My earlier "45–70 tok/s" was from the chat UI; the engine's sustained averages are 44–52 tok/s. The 70 figure was a short burst.