#5653 [Engine] SYCL (Arc): systematic instruction-following gap vs CUDA on identical weights — 2 % vs 77 % on paired probe

open · @mikellantalde · 5 commentaires · Sur GitHub

Setup & installAMD / HIPNVIDIA / CUDAModels & quantsSecurity

Description


**Strata version:** SYCL build from main @ `9f7d977d` · CUDA reference 0.1.56
**Graphics card, memory and system:** Intel Arc B70 32 GB · 62 GB RAM · Ubuntu 26.04 LTS (CUDA reference box: RTX Pro 6000 Blackwell 96 GB, used with a 28 GB VRAM cap — shared card · 125 GB RAM)
**Model, size and settings:** Qwen3.8-Flash-Next UD-Q4_K_XL (pack v31q8) · 260K context · `--kv int8` · full `--native` preset · temperature 0 · thinking medium
**Reproduction status:** reproduced locally on both boxes, repeatedly
## Summary
We run the same model pack and the same runtime flags on two boxes: one SYCL
(Intel Arc B70, 32 GiB) and one CUDA reference (RTX Pro 6000 Blackwell, used
with a 28 GiB VRAM cap). At **temperature 0** the SYCL build
scores ~5 points lower on IFBench (53.7 vs 59.0 strict, n=300 each), and a
paired probe shows the gap is **not noise**: on the 28 prompts the CUDA box
passed and the SYCL box failed twice, the SYCL box passes **2/84 re-draws
(2 %)** while the CUDA box passes **43/56 (77 %)** across two draws
(two-proportion z ≈ 9.2). The failures are systematic and of one kind: the
SYCL box writes fluent, on-topic answers but **drops the global mechanical
constraint** (indentation stairs, alphabet word order, exact number counts,
periodic-position words). Independently, the SYCL box shows a low-rate
**language drift** (a second language bleeding into agent tool-loop
narration) that we have never observed on the CUDA box at any context length
up to ~190 K tokens with the same weights.
All configuration-level differences have been eliminated by direct
experimentation; the remaining divergence appears to live in the parts of the
port that have no CUDA-equivalent implementation (see *Where the source
points*). This is not a complaint — the port is otherwise excellent and the
box is our daily driver — it is a bug report with measurements, and we are
happy to run any paired dump you want.
## Setup
| | SYCL box | CUDA box (reference) |
|---|---|---|
| Engine | built from Strata main @ `9f7d977d` (SYCL port) | release build 0.1.56 |
| GPU | Intel Arc B70 32 GiB | RTX Pro 6000 Blackwell (28 GiB cap, shared) |
| Model | identical pack: Qwen3.8-Flash-Next, UD-Q4_K_XL, pack `v31q8` | same |
| Flags | `--native <shard1> --kv int8 --max-context 260000` etc. | same family, MTP spec |
| Drafting | DFlash2 spec 6 (`--draft-conf 0.6`) | MTP spec 4 |
| Sampling | temperature 0 everywhere in this report | same |
Both engines run the **full `--native` preset** (it auto-enables
`native_bf16(-extra)`, `native_qsa`, `native_qsa_indexer`, `native_gdn`,
`native_rope`, `native_moe_combine`, `native_router`, `native_ple_key`,
`gr_native_mmvf` — `generate.cpp:2505-2509`), so the entire `--native-*`
surface was already active for every measurement below.
## Ruled out by experiment (each measured, not guessed)
| Lever changed on the SYCL box | IFBench strict result |
|---|---|
| PLE table IQ4 → Q8 (`--ple-gguf`) | 53.7 → 53.7 (unchanged) |
| Router/SSM/indexer BF16 oracle (`--native-bf16 --native-bf16-extra`) | 53.7 → 53.7 |
| `--pcie-frac 1` (all streamed experts on GPU) | unchanged; gap persists |
| Full re-run with all three at once | 53.7; 25/300 prompts flipped +12/−13 → noise band ≈ ±1 pt |
Also ruled out **by code** (so nobody spends time on it): draft acceptance at
t=0 is lossless — the verify loop accepts a draft only on exact match with the
target argmax (`generate.cpp:11827`): `while (a < A.T - 1 && A.tok[a+1] ==
outp[a]) ++a;`. `--draft-conf` is a performance gate, not a verdict. Greedy
output is pure target greedy on both boxes.
## The paired probe (the evidence)
Selection: prompts from the public IFBench test set that the CUDA box passed
(draw 1) and the SYCL box failed on **two independent draws**. 28 prompts
(count 9, words 7, format 6, ratio 3, repeat 2, custom 1). Then: SYCL re-drew
3× total, CUDA 2× total, temperature 0, thinking medium, official IFBench
verifier.
| Box | Passed | Rate |
|---|---|---|
| CUDA | 43/56 | 77 % |
| SYCL | 2/84 | 2 % |
(Selection bias works *against* the CUDA number on its second draw — it drops
to 54 % there — these items are hard for both; the asymmetry is the point.)
Failure examples (public test prompts; full 28 attached as
`probe-28-systematic.jsonl`):
* *"Create stairs by incrementally indenting each new line…"* — CUDA indents
  the three lines; SYCL returns three unindented lines.
* *"Each word in your response must start with the next letter of the
  alphabet…"* — CUDA walks a→z correctly; SYCL breaks at word 3
  ("Anime bursting **with**…" — expects `c`).
* *"Include exactly 5 numbers…"* — CUDA emits exactly 5; SYCL writes correct
  content but **injects extra numerals in the prose** ("4 main components",
  "3 essential functions").
The pattern is consistent across families: comprehension intact, style fine,
**global constraints dropped** — precisely the function that long-horizon
instruction retention depends on.
## Where the source points (same weights → per-token numerics is all that is left)
With the full `--native` preset active, three per-token paths remain
SYCL-specific by the port's own admission:
1. **QSA top-k selection**: `sycl/src/kernels/cuda/qsa.dp.cpp:6` —
   *"THE THRESHOLD SEARCH IN `topk_512` RATHER THAN A SORT. The selection has
   to reproduce `ggml_top_k`"* — a hand-rolled threshold search (+1e9
   masking, tie handling) standing in for a sort. Block-membership flips at
   the cut boundary change what a token attends to — the exact function the
   failure signature points at.
2. **Hyper-connection read**: `sycl/src/kernels/cuda/fused_gr.dp.cpp:2141` —
   *"SYCL port: the q8_1 images (S26 STRATA_QFUSE) and the Q8_0 reads
   (S23 STRATA_HC_Q8) are not built; the caller quantizes"* — CUDA reads the
   stored Q8_0 directly; SYCL requantizes. (This also explains why we measured
   `STRATA_HC_Q8=1` as a no-op: the consumer is not built.)
3. **Attention kernels**: split-K decode/prompt attention are the port's own
   (`qsa_decode_attn.dp.cpp`, `qsa_prompt_attn_*.dp.cpp`,
   `g_fast_attn`/`native_flash_attn_short` in `layer.cpp:540-548`).
Related threads (adjacent, not duplicates): #5198 reports the QSA indexer
budget (`idx_top_k = 2048`, hard-pinned in the score kernel — the same line
we quote above) as a source of degraded instruction-following at depth on a
CUDA box: different axis (budget vs build parity), same subsystem, and
corroboration that this subsystem governs exactly the task class where our
two builds part ways. The SYCL CPU VNNI tier's rounding differences (PR
#5481, #2713) are pre-empted here: our measurements run with `--pcie-frac 1`,
so CPU-side expert kernels are off the per-token path in every number above.
## Language drift (secondary, same substrate)
Sampling completed agent transactions every 15 min on the SYCL box:
**0.2–0.9 % of samples** contain short second-language (Galician) bleed into
tool-loop narration ("agora", "ficheiro", …). Occurred at contexts from
**15.9 K to 151.6 K**, across every configuration above (incl. full
`--native`, Q8 PLE, `--pcie-frac 1`). The CUDA box, under heavier real traffic
and the same weights, has shown **zero** such episodes. We mention it because
it is consistent with the same systematic per-token divergence and because
two symptoms are easier to reproduce than one.
## Reproduce
1. Take the 28 prompts attached (`probe-28-systematic.jsonl`, public IFBench
   test prompts + instruction ids).
2. Query both engines (or just the SYCL one) at `temperature 0`,
   `reasoning_effort medium`, `max_tokens 2048`, chat completions, then score
   with the official IFBench verifier (`run_eval.py`).
3. Expected: SYCL ≈ 0–2/28; CUDA ≈ 15–22/28 per draw.
4. A ~60-line driver is embedded below.
<details>
<summary>probe_gen.py</summary>
```python
import argparse, json, os, time, urllib.request
AP = argparse.ArgumentParser()
AP.add_argument('--test-file', default='probe-28-systematic.jsonl')
AP.add_argument('--base-url', required=True)   # e.g. http://<host>:<port>/v1
AP.add_argument('--model', required=True)
AP.add_argument('--api-key', default=os.environ.get('PROBE_API_KEY', ''))
AP.add_argument('--tag', required=True)
A = AP.parse_args()
os.makedirs('out-' + A.tag, exist_ok=True)
test = [json.loads(l) for l in open(A.test_file, encoding='utf-8')]
with open('out-%s/%s-responses.jsonl' % (A.tag, A.tag), 'w', encoding='utf-8') as out:
    for t in test:
        body = {'model': A.model,
                'messages': [{'role': 'user', 'content': t['prompt']}],
                'temperature': 0.0, 'max_tokens': 2048,
                'reasoning_effort': 'medium', 'stream': False}
        req = urllib.request.Request(
            A.base_url.rstrip('/') + '/chat/completions',
            data=json.dumps(body).encode(), method='POST',
            headers={'Content-Type': 'application/json',
                     'Authorization': 'Bearer ' + A.api_key})
        try:
            ch = json.loads(urllib.request.urlopen(req, timeout=1800).read())['choices'][0]
            c = (ch['message'].get('content') or '').strip()
        except Exception as e:
            c = ''
            print(t['key'], 'ERROR', repr(e)[:120], flush=True)
            continue
        json.dump({'prompt': t['prompt'], 'response': c}, out, ensure_ascii=False)
        out.write('\n')
        print(t['key'], 'ok', len(c), flush=True)
```
</details>
## Questions / asks
1. What is the intended parity contract for the `topk_512` threshold search
   vs `ggml_top_k` — is there a known tie/epsilon regime where they diverge?
2. Are the fused-GR Q8_0 reads (`STRATA_HC_Q8`) on the port roadmap, or is the
   caller-quantize path intended to be permanent?
3. Would a paired `--dump-logits` capture on one hard prompt (last-token logit
   vectors, CUDA vs SYCL) help you bisect — attention vs selection vs HC? We
   can run it at our next scheduled restart window and attach the diff.
Happy to provide further paired measurements. The SYCL box is our daily
driver and we would rather land this as a fix than live with the gap.

[probe-28-systematic.jsonl.txt](https://github.com/user-attachments/files/34574904/probe-28-systematic.jsonl.txt)

Sur le site

Liens install, modèles, releases.