Pull requests / #1764

#1764 EXL3: run the EXL3 trellis format on the engine

open · @Yxmura · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Description

Adds a loader and a fused decode path for EXL3 (ExLlamaV3's trellis-coded weight format). Integer K = 1..8, all three codebooks (mul1/mcg/3inst). K is read per tensor from the trellis shape, so every turboderp/Qwen3.8-Flash-Next-exl3 branch runs.

## What

- **docs/EXL3.md, tools/exl3/, tools/test_exl3_*.py.** Format spec, numpy codec (codebook, reconstruct, ngram), safetensors/tokenizer adapters.
- **CPU kernel** `src/kernels/cpu/exl3.cpp`. Decode LUT, sliding window, tensor-core perm, H128, folded GEMV, ngram row.
- **GPU kernels** `src/kernels/cuda/exl3.cu`, `exl3_experts.cu`. Reconstruct, fused decode-GEMV, batched MoE expert FFN, `Exl3ExpertStore`.
- **Loader** `include/strata/artifact/{safetensors,exl3_model}.hpp`.
- **ngram** `PleTable::open_exl3`.
- **Engine seam** `WeightRef::exl3` + `gemv_quantized` (`src/core/layer.cpp`).

## Bitrates

| branch | experts | attn / head |
|---|---|---|
| 2.05bpw_h4_ng4 | K=2 | K=4 |
| 3.05bpw_h5_ng5 | K=3 | K=5 |
| 4.05bpw_h6_ng6 | K=4 | K=6 |
| 5.05bpw_h6_ng6 | K=5 | K=6, K=7 |
| 6.05bpw_h6_ng6 | K=6 | K=8 |

## Measured

Radeon PRO W7800 (gfx1100, ROCm 7.13), real weights, 10 experts resident.

| | |
|---|---|
| expert FFN | 0.366 ms/layer, ~57 tok/s expert-compute-bound (x48 layers) |
| layer 0 / 47 vs CPU | rel 1.1e-3 / 1.0e-3 |

## Validation

- `ctest -R exl3`: 20/20. CPU numpy fixture + parity per K=1..8; `exl3_gpu_parity` K=1..8; `exl3_gemv_parity` K=1..8 plus the expert (640x2560) and attn (2560x10240) shapes; `ple_exl3_parity`.
- `python3 tools/test_exl3_{codebook,reconstruct,ngram}.py`: OK.
- Full `strata` target builds.

Sur le site

Liens install, modèles, releases.