Pull requests / #168

#168 hip: support Q2_K embeddings and type-aware PLE table rows

closed · @samuelishida · 0 comments · View on GitHub

AMD / HIPNVIDIA / CUDAModels & quants

Description

Extends the HIP/CUDA quant paths to cover more public checkpoints:

- Bartowski's IQ2_XS uses Q2_K for `token_embd.weight`; add a GPU dequantizer and embedding gather, exposed through `iq_dequant_supported()` so it is accepted for embeddings but not silently treated as an MMVQ type.
- The AD IQ4_XS-M64 checkpoint stores PLE table rows as Q5_1; make row decode type-aware for Q4_0 / IQ4_NL / Q5_1 with matching row-byte counts in `PleReader` and its row cache.
- Add `hip_q2k_embed_parity` to exercise both synthetic and optional real IQ2_XS rows.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.