Pull requests / #1800

#1800 Support external PLE replacement for a missing split shard

open · @constantindjonkam · 0 comentarios · En GitHub

NVIDIA / CUDAModels & quants

Descripción

## What changed

- Add opt-in `--skip-ple-shard N` support for split GGUF models when an external `strata-ple` file is supplied with `--ple-gguf`.
- Keep default split validation strict: unrelated missing shards and inconsistent tensor counts still fail.
- Validate the external PLE format and model geometry; when the skipped file is present, validate its header as the model's single PLE tensor.
- Thread the omission through native embedding/head loading and `tools/iq_pack.py`.
- Document the Swift Q4_K_L packing invocation.

Example:

```bash
python tools/iq_pack.py \
  --gguf Swift-...-00001-of-00004.gguf \
  --out packs/swift-q4kl \
  --ple-gguf ple_fp8.gguf \
  --skip-ple-shard 2 \
  --compat-bf16
```

The packer does not write `experts.bin` unless explicitly requested.

The replacement interface is described generically as an external PLE artifact; the current example and validator use the existing FP8/E4M3 PLE representation and still validate its format and model geometry.

## Validation

- `gguf_split_test`: passed, including accepted PLE-only omission and refused unrelated omission.
- `python tools/test_iq_pack.py`: 25 passed.
- CUDA Release build with MMQ/Q6 support: `strata` and `gguf_split_test` built successfully.
- Packed the real Swift Q4_K_L model without `--experts-bin`.
- Loaded and generated one token with the external FP8 PLE while shard 00002 was absent from the runtime directory (RTX 3090, GPU 0 only); output was produced successfully.

En el sitio

Enlaces a install, modelos, releases.