Pull requests / #2911

#2911 prefill: a fixed --prefill above 65280 tokens is capped, 65536 stopped the engine at the first longer prompt (#2743)

open · @ischencheng · 0 Kommentare · Auf GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

Fixes #2743.

With a fixed `--prefill 65536`, the first prompt longer than 65536 tokens stops the engine with `iq_embed_rows: invalid argument`. The prompt path launches some kernels with one grid row per token of the chunk, and a grid's y is at most 65535. The token embedding is the first one, the GDN q/k norm in `gdn_conv` (`gdn_l2_kernel`) is the next.

This caps a named `--prefill` at 65280 (the largest multiple of 256 under the limit) and prints one line at start. `auto` never gets there (32768 at most), and named chunks up to 65280 are unchanged, so default starts are the same.

I also tried tiling those two launches instead. That ran the 65536 chunk on the setup below, but other launches take one row per token too (for example the peer gather in `src/prefill/kernels.cu`), and I could only test one pack and one card. A cap in one place covers paths I could not run. Happy to switch to the tiling version if you prefer it.

One L40S 48 GB (sm_89, Linux), IQ2_XS native pack, `--max-context 131072 --kv int8`, upstream main 477ac13c. Each start reads the same 70,433-token prompt with two codes in it, one in each chunk, then a short chat:

| build | `--prefill` | prompt path loan | 70K prompt | reply |
|---|---|---|---|---|
| main | 65536 | 21323 slots | engine exits, `iq_embed_rows: invalid argument`, HTTP 503 | - |
| main | 65280 | 21242 slots | 4422 tok/s | `BLUE-4417, RED-9052` |
| this PR | 65536 (capped to 65280) | 21242 slots | 4362 tok/s | `BLUE-4417, RED-9052` |

The capped start runs the same as main at 65280: same loan, same replies, same draft counts. The two speeds are single runs.

`generate.cpp` with this change also compiles for HIP (gfx1201, ROCm 7.0), same warnings as main.

Not tested: Windows, the NVFP4 pack from the report, HIP on a GPU, a layer split. SYCL has its own `generate.cpp` and is not touched; I don't know its limit there. Qwen3.6 (`strata-q36 --chunk`) is a separate program and not touched.

Mehr auf der Site

Links zu Install, Modellen, Releases.