Issues / #1843

#1843 Engine outputs only token 0 on RTX 4090 (sm_89, locally built 0.1.41) with IQ3_XXS GSQ-RCO; bundled llama.cpp runs the same GGUF fine

open · @wannagoxuf · 2 comentários · No GitHub

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quants

Descrição

### Summary

On a locally built 0.1.41 engine (sm_89), `generate` and `--serve` on Qwen3.8-Flash-Next IQ3_XXS (ISTA-DASLab GSQ-RCO) output **only token id 0** for every position, on one GPU and on a 4-GPU layer split, with `STRATA_MMVQ_IL=0` or unset. The **same GGUF runs fine with the bundled llama.cpp** (third_party/llama.cpp, CPU build) — coherent output, both English and Chinese.

### Environment

- 4× NVIDIA GeForce RTX 4090 (24 GB, sm_89), EPYC 9274F, 566 GB RAM, Ubuntu 24.04
- Driver: kernel module 580.173.02, user space matched to 580.173.02 (the 580.178 update is loaded but the module cannot be reloaded without a reboot; libcuda/libnvidia-ml are aligned to 580.173.02 for all runs below)
- CUDA toolkits: /usr/local/cuda 12.2 and 12.8
- Strata: git main at fb58e0d (0.1.41), `./setup.sh --yes --family qwen --model IQ3_XXS --vision gpu --gpus 0,1,2,3 --data-dir ... --no-start --source modelscope`
- Engine compiled locally, `CMAKE_CUDA_ARCHITECTURES=89`

### What happens

```
$ engine/strata --gpu 0 --pack .../packs/iq3_xxs --native .../IQ3_XXS-00001-of-00002.gguf \
    --ple-gguf .../IQ3_XXS-00002-of-00002.gguf --expert-profile data/expert-profile.bin \
    --expert-cache auto --prefill 2048 --spec 2 --max-context 4096 --kv int8 \
    --tokens "760,62865,1020,47358,284" --max-new 8
prompt  : 760 62865 1020 47358 284
output  : 0 0 0 0 0 0 0 0
```

`--serve` shows the same through the API (every reply is `!!!!…`, i.e. token 0 repeated). Speeds are normal (prefill ~7-11k tok/s at 32K, decode ~200 tok/s single GPU).

### What I ruled out

- **GGUF data**: sha256 of both shards matches Hugging Face exactly (`219ea929…`, `316b46f3…`). (The download was damaged in transit by CDN range-response mismatches; I verified and repaired every 512 MB segment against the origin, then re-verified the full-file hashes.)
- **Quantization declarations**: the pack's `native_experts.txt` per-layer `gu_type/d_type` match the GGUF tensor headers (the GSQ-RCO files use per-layer mixed types: IQ2_XS / IQ2_XXS / IQ4_NL / Q2_0 / IQ3_XXS / IQ3_S / Q5_K / Q4_K / IQ4_XS; e.g. layer 0: ffn_up/gate_exps IQ2_XS, ffn_down_exps IQ4_NL).
- **Kernels**: `ctest` 97/101 pass on this machine, including `mmvq_il_parity` (bitwise, all cases), `kv_stream_parity`, `qsa_*_parity`, `gdn_rec_parity`, `gemm_bf16_parity`, `dequant_s2_parity`, `iq_parity`, `ple_reader_selftest`. The 4 failures are missing-test-data issues (ple_parity wants a Q2_0 shard 2; expert_parity/pool_test want pack/full/experts.bin; platform_memory_test is RAM-state dependent), not engine mismatches.
- **Layer split**: single GPU (`--gpu 0`) reproduces it; so does 4-GPU split `[12,24,36]`.
- **MMVQ IL path**: `STRATA_MMVQ_IL=0` — unchanged.
- **Upstream**: local HEAD is the current main tip (fb58e0d), nothing to pull.

### The suspicious lead: the native head's type

Single-GPU runs print:

```
strata generate: experimental native Q5_K head, 437043200 bytes, in 0.1 s
```

But this GGUF's `output.weight` is **BF16** (2560 × 248320 → ~1.27 GB as BF16; 437043200 bytes is the Q5_K layout size). I did not find a place where a BF16 output.weight should be loaded as a Q5_K-layout native head. In the 4-GPU case the head takes the canonical path and the output is still token 0, so this may be one of two independent problems — or a red herring for the split case.

### Side finding

`--no-fused-gr` (single GPU) hangs the generate run at the verify-window stage (no output, no exit; killed after 10 min). The fused path itself runs and completes.

### Request

Pointers on where to look next (or an existing known issue) would help. Happy to run additional diagnostics, A/B flags, or provide full logs — the machine is reproducible in minutes.

No site

Links install, modelos, releases.