#2685 Strata serves Qwen3.8-Flash-Next with strange behavior - really bad results

Description

# Strata serves Qwen3.8-Flash-Next dumber than llama.cpp: tool-call loops + engine nondeterminism

Hardware: RTX 5070 12 GB, 60 GB RAM (only one 46 GB instance fits at a time), driver 595.99.02, CUDA 13.2.
Pack: IQ3_S (`ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:IQ3_S`), Strata native pack.
Reference server: `llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:IQ3_S -c 110000 -np 1
-fa on -ctk q8_0 -ctv q8_0 -b 4096 -ub 1024 --load-mode none --no-mmproj --port 8080 --cpu-moe --moe-cache-mib 2048`

## 1. Observed behavior (the complaint)

Same model, same prompts, two servers:
- Via Strata: tool calls fail most of the time (malformed args, hallucinated tools:
  `Bash`/`exec`/`WebFetch`/`registry`/`search`, OpenCode-style args like `file_path`, `limit`/`offset`
  that don't exist in the offered schema). Failures trigger agent retry loops. Also "dummy loops":
  very short thinks followed by the same command over and over (`list_files` x3-x5, `ls -la` x5).
- Via llama.cpp (`llama serve`, same quant): noticeably better reasoning, working tool calls.
- Result: Strata not usable as daily driver.

## 2. Issues found AND fixed (config)

**2a. Greedy-by-default sampling.** `strata-iq3_s.json` had no `sampling` block, and
`serve/server.py` only forwards `temperature>0` / `top_p<1` etc, so the engine ran greedy
(temperature 0). Greedy decoding is the classic cause of repetition loops.
**Fix applied** to `strata-iq3_s.json` (backup `strata-iq3_s.json.pre-fix.bak`):
`sampling {temperature 0.6, top_p 0.95, top_k 20}`.

**2b. `tool_call_recovery` off.** Absent from config = False (`server.py:3102,6864`;
`docs/DETAILS.md:845` quotes 92.9% vs 98.3% tool-call success on 1462 turns).
**Fix applied**: `tool_call_recovery true`.
Parser proof (GPU-free): synthetic 5-shape check 1/5 recovered without the flag, 5/5 with it;
`serve.test_server.ToolCallRecovery*` 12/12 pass.

## 3. Agentic evals (controlled, same harness)

Harness (`evidence/dailydriver*.py`): agentic loop over a slide-generator repo
(`/home/gustavo/Testes/lol/slide_test`: ppt.js, deck_claude.js, styles.css, pdf_text.txt,
binary pptx/pdf), tools `list_files`/`read_file`/`grep`, full call trace, keyword scoring.
- v1 (5 runs/arm, max_tokens 1024): 0/10 summaries, mostly `finish=length` (harness flaw: 1024
  starves xhigh thinking).
- v2 (max_tokens 4096, 8 turns, 5 runs/arm): old-greedy arm loops 4/5, fixed arm loops 2/5 +
  hallucinations 2/5. **Both arms 0/10 task completions.**
- Same harness vs llama.cpp (arm C, port 8083): **5/5 clean, 14-15/15 valid calls, zero loops.**
- v3 (fixed config, temp 0.6, 3 runs): **0/3 usable** — same disease (list_files loops, binary reads,
  hallucinated `write_file`). Config fix did NOT cure it.
- Ruled out prompt rendering: Strata `chat_template.jinja` byte-identical to llama.cpp `/props`
  template (183 lines, empty diff); Strata-rendered vs llama-tokenized prompt ids IDENTICAL
  (516/516 turn 1, 562/562 turn 2, tools block included). Difference is engine-side (logits)
  and/or sampling RNG (harness sends `seed`; Strata seed-honoring untested).
- Single-request continuation test: Strata 1/6 full reads + Bash hallucination + partials;
  llama 6/6 correct progress.
- Near-greedy (temp 0.01, seeds 111x2+222): llama 3/3 bit-identical; Strata nondeterministic
  **with the same seed**. <-- this redirected the whole investigation to the engine.

## 4. Engine nondeterminism hunt (the deep part)

Method (`evidence/run_determ.sh`): canonical engine args (pack iq3_s, native+ple GGUFs,
expert-profile.bin, cache auto, prefill auto, spec 4, mtp, max-ctx 110000, kv int8, pcie-frac 0,
spec-min-p 0.75, vram-reserve 896) + `--tokens-file prompt609.txt` (609 ids) + `--max-new 24` +
`--window-hashes`. Two fresh processes, compare hashes. Baseline: DIFFER from prefill-tail
window 0 (residual hash 2df94610 vs 8860bf2c, argmax 9764 vs 40; 20 vs 26 windows).

Still differed after each of (one factor at a time): MTP removal, `--pool-workers 1`, reasoning
budget, `--prompt-cache 0`, `STRATA_ROUTE_TAIL_SKIP=0`, `--no-coupled-draft`,
`STRATA_PREFILL_CPU_SHARE=0`, `--sync-every-layer`, `--adapt-every 0`,
`--expert-cache-cpu-order`, `--ple-io ram`, `--no-host-worker`, `STRATA_PREFILL_ISSUER=0`,
`STRATA_PF_STEP_SYNC=1`, `--pcie-frac 0` (partial: window-0 argmax identical, diverges token 3).
`--dump-routing`: records 0-2 identical, record 3 differs. `--dump-logits` writes empty files
(broken); `--dump-residual` all zeros; `--dump-layers` OOMs (~1.2 GB dump buffers).

Per-layer ground truth via the research hook `STRATA_DBG_FEAT` (+`_LAYERS`, `prefill.cpp:2948`;
parser `evidence/featparse.py`): layers 0 AND 1 bit-identical (ids, weights, hidden states),
layer 2+ differ hugely (~1.5M/1.55M values, magnitudes ~0.9-1.7). Onset varies run to run
(layer 1 vs 2). Layer-1 MoE fp32 input bit-identical. **Decisive: `STRATA_PREFILL_MMQ=0` runs are
bit-IDENTICAL** (hashes byte-equal, streamed/resident counts exactly equal) => the divergence
is inside the MMQ expert-compute path; attention/routing/tokenizer/server/sampling cleared.

Cleared by experiment: same-process back-to-back greedy server requests DIFFER (live race, not
just process state); VRAM-poison P0-vs-P1-vs-P0b all differ (not uninitialized memory);
`CUDA_LAUNCH_BLOCKING=1` still diverges at layer 2 (not a GPU overlap race; prefill completes,
then hangs at verify capture — known issue #1341); CPU-share=0 still diverges (pool truly idle:
`PoolHold` gating, `cpu_maybe`); CLB+SHARE=0 still diverges (both jointly cleared);
stream_all-forced path still diverges (STAGE-ring cleared); tiny prompts T=2,4,8 all diverge
(shape theory dead); near-all-streamed (`--expert-cache 32` + SHARE=0) still diverges;
no-MTP (`--spec 2`, `+0 MiB for the draft head`) still diverges (draft cleared).
Cleared by audit: staging-slot protocol, upload rings (all single-stream `m.cs`), host grouping
tables (per-layer full sync), stager + file sources (thread-safe libaio/OVERLAPPED),
static residency (`host_res` built once at startup; full 48x512 table verified identical),
no float atomics anywhere, combine (same call both modes), vanilla unmodified llama.cpp MMQ
headers (third_party git-clean), deterministic J-loop / stream-K / launch selection.

## 5. The paradox (where the hunt stands)

Custom uncommitted debug patch `STRATA_DBG_MOE_TENSORS` (`prefill.cpp` ~4082) dumps layer-1
Xq/GU/Dm/bo + gathered blobs + bounds + cnt/off + residency + row map. Across runs, EVERY
gate/up-product input is bit-identical (activations, all 16 group blob slots, device bounds
nbound=957, counts, offsets, static residency, ids 0..N) yet gate/up products differ on ALL
6080 rows (resident and streamed experts alike). Standalone replay harness
(`evidence/replay.cpp`): the kernel is deterministic in isolation (15/15 identical) — but its
bits don't match the engine's (fidelity gap; the pack's actual gate/up type value is unverified).
compute-sanitizer synccheck hung at the #1341 verify capture and was killed (inconclusive).

## 6. What persists / next steps

- Root cause UNKNOWN. Air-tight paradox: identical inputs + deterministic kernels + different
  outputs is logically impossible, so one verified claim must be false (fidelity gap in what the
  engine actually launches) or the sm_120 kernel build misbehaves. Next: close the replay
  fidelity gap (actual `gu_type`, launch params), then per-group launch-param dump.
- Workaround with evidence: `STRATA_PREFILL_MMQ=0` is the only flag that gave bit-identical
  runs. Agent-quality validation (3x repo test, fixed config + MMQ=0) was running when this was
  written — see `evidence/dailydriver4.jsonl` / `dailydriver4.log`, results appended below.
- Still owed regardless: greedy temp-0 3x test (workaround check), Strata seed-honoring check.

## Appendix A. Validation: MMQ=0 as daily-driver workaround

Apples-to-apples with v3 (B-fixed config, temp 0.6, same repo/task/harness, 3 runs), plus
`STRATA_PREFILL_MMQ=0` in the server environment (`evidence/dailydriver4.py`).

### Results (runs completed 2026-10-15)

| run | turns | tool calls | executed | loop | empty turns | think chars | SUMMARY | kw/10 |
|-----|-------|-----------|----------|------|-------------|-------------|---------|-------|
| 0 | 8 | 14 | 14 | no | 0 | 1785 | none (still exploring at turn cap) | 0 |
| 1 | 8 | 14 | 14 | no | 0 | 1625 | none (still exploring at turn cap) | 0 |
| 2 | 8 | 13 | 13 | no | 0 | 1740 | none (still exploring at turn cap) | 0 |

All 41 tool calls executed (no parse failures, no hallucinations: no `Bash`/`exec`/
`WebFetch`/`registry`, no OpenCode-style `file_path`/`limit`/`offset` args). Every run did
productive exploration — `list_files`, then `read_file` of all four real repo files
(`deck_claude.js`, `ppt.js`, `pdf_text.txt`, `styles.css`) plus targeted `grep`s
(`writeFile`, `SLIDE \d+`, `FONT_*`, `MASTER_`, ...). Zero dummy loops, zero empty turns.

Verdict: **MMQ=0 cures the loop/hallucination disease** (v3 with MMQ on: 0/3, dummy
`list_files` loops + hallucinated tools/args). But **task completion is still 0/3** by the
harness metric: no run produced a SUMMARY within the 8-turn cap (all were still gathering).
Caveats: (a) server stdout/stderr went to DEVNULL, so no prefill-timing numbers and MMQ=0
is confirmed only via env inheritance (the same mechanism as the bit-identical NOMMQ engine
runs); (b) 8 turns may be too tight — the MMQ=0 runs spent their turns reading instead of
looping, so a longer cap or a nudge-to-summarize might complete; (c) determinism does not
imply quality — this validates stability, not that summaries are good.

## Appendix B. Evidence files in this repo

- `evidence/run_determ.sh` — engine A/B harness (canonical BASE args)
- `evidence/prompt609.txt` — the 609-token prompt (comma-separated ids)
- `evidence/featparse.py` — FEAT dump parser (proves layer-0/1 identical, 2+ differ)
- `evidence/dailydriver2.py`, `dailydriver3.py`, `dailydriver4.py` — agentic-loop harnesses
  (+ `.jsonl` traces, `.log` outputs)
- `evidence/replay.cpp` — standalone MMQ-kernel replay (15/15 deterministic in isolation)
- `evidence/armB.json` — fixed server config used by the harnesses
- `evidence/NOMMQa.txt`, `evidence/NOMMQb.txt` — the byte-identical MMQ=0 window hashes
- `evidence/W1.txt`, `evidence/W2.txt` — baseline differing window hashes
- `evidence/sp_req.py`, `evidence/poison.py` — same-process probe, VRAM-poison tool

[dailydriver4.log](https://github.com/user-attachments/files/33513291/dailydriver4.log)
[TINY8b.txt](https://github.com/user-attachments/files/33513285/TINY8b.txt)
[TINY8a.txt](https://github.com/user-attachments/files/33513284/TINY8a.txt)
[N2.txt](https://github.com/user-attachments/files/33513279/N2.txt)
[N1.txt](https://github.com/user-attachments/files/33513275/N1.txt)
[W2.txt](https://github.com/user-attachments/files/33513280/W2.txt)
[W1.txt](https://github.com/user-attachments/files/33513273/W1.txt)
[NOMMQb.txt](https://github.com/user-attachments/files/33513287/NOMMQb.txt)
[NOMMQa.txt](https://github.com/user-attachments/files/33513283/NOMMQa.txt)
[armB.json](https://github.com/user-attachments/files/33513289/armB.json)
[replay.cpp](https://github.com/user-attachments/files/33513277/replay.cpp)
[poison.py](https://github.com/user-attachments/files/33513274/poison.py)
[sp_req.py](https://github.com/user-attachments/files/33513281/sp_req.py)
[prompt609.txt](https://github.com/user-attachments/files/33513288/prompt609.txt)
[dailydriver4.py](https://github.com/user-attachments/files/33513278/dailydriver4.py)
[dailydriver3.py](https://github.com/user-attachments/files/33513286/dailydriver3.py)
[dailydriver2.py](https://github.com/user-attachments/files/33513276/dailydriver2.py)
[featparse.py](https://github.com/user-attachments/files/33513282/featparse.py)
[run_determ.sh](https://github.com/user-attachments/files/33513290/run_determ.sh)