Pull requests / #168
#168 hip: support Q2_K embeddings and type-aware PLE table rows
closed · @samuelishida · 0 Kommentare · Auf GitHub
AMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
Extends the HIP/CUDA quant paths to cover more public checkpoints: - Bartowski's IQ2_XS uses Q2_K for `token_embd.weight`; add a GPU dequantizer and embedding gather, exposed through `iq_dequant_supported()` so it is accepted for embeddings but not silently treated as an MMVQ type. - The AD IQ4_XS-M64 checkpoint stores PLE table rows as Q5_1; make row decode type-aware for Q4_0 / IQ4_NL / Q5_1 with matching row-byte counts in `PleReader` and its row cache. - Add `hip_q2k_embed_parity` to exercise both synthetic and optional real IQ2_XS rows.
Mehr auf der Site
Links zu Install, Modellen, Releases.