#6102 [Engine] use runtime expert count in layer-major prefill

closed · @soypsi · 0 Kommentare · Auf GitHub

Setup & installAMD / HIPNVIDIA / CUDAModels & quantsSecurity

Beschreibung

## Problem and result

Layer-major prefill uses the compiled `NE = 512` as the model's expert count. For packs with fewer experts (including Coder IQ1_M with 256 and a reported 384-expert pack), it can schedule nonexistent experts and index the resident-expert table with the wrong stride, aborting long prompts. The layer-major path now consistently uses `m.g->n_expert` for job sizing, bounds, loops, and resident indexing.

## Related discussion

- #5104 — original Coder IQ1_M long-prompt abort
- #5966 — newer PF_SMALL report with the same invalid expert IDs

## Approach

Use the model's runtime expert count throughout `Prefill::lm_run`. For 512-expert models this remains the existing count; smaller packs no longer create jobs for nonexistent experts or use an incorrect resident-table stride.

The implementation was prepared by GitHub Copilot CLI. @soypsi reproduced the failure and tested the rebuilt engine in real use; they did not author this code change.

## Validation

**Passed:**

- `cmake --build build --target strata -j2` — passed on current upstream `main` (`c77572806996349b49995e4f98d41b5294d36a21`) plus this change; CUDA 13.1, `sm_89`; executable linked successfully.
- `./update.sh --draft-vocab en` — previously rebuilt the user's installed engine (0.1.59.1); the previously failing long-context request then succeeded. This runtime check was on the earlier 0.2.13 source, not this current-main revision.
- Independent before/after hardware reports on the patch are in #5104: a 256-expert test (comment by Norhiro) and a 384-expert test (comment by kaban-baku).

**Failed:**

- None observed.

**Not run:**

- Current-main long-context runtime test on hardware; the local build is compile-only evidence for this revision.
- HIP, SYCL, and Metal builds. The available build is CUDA-only; these checks need suitable toolchains/hardware.
- The repository test suite; `STRATA_BUILD_TESTS` is off in this build configuration.

## Limits

This remains a draft pending current-main hardware validation and the unavailable backend builds. Workaround `STRATA_PF_SMALL=0` is reported in #5966, but upstream `main` still contains the `NE` assumptions and no competing fix PR was found.

Mehr auf der Site

Links zu Install, Modellen, Releases.