Issues / #2743
#2743 iq_embed_rows: missing the >65535-row guard - fixed --prefill 65536 fails deterministically at a 64K chunk (invalid argument)
open · @daifeiya · 0 comments · View on GitHub
BenchmarksServer & APINVIDIA / CUDAWindows
Description
Environment: Windows 11, RTX 4090 48GB (non-D), driver 617.42, engine/project 0.1.44.1 (source tree), NVFP4 pack (Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata).
Symptom: with a FIXED `--prefill 65536`, every launch fails when the 131072-token tier is requested: the engine's last log line is `iq_embed_rows: invalid argument`, the request returns HTTP 503, and the engine stops serving (the Python server stays up). Reproduced 3/3 launches (deterministic). The same config completes every tier up to 65536 (actual 65527 tokens).
Log excerpt (run 1 of 3, tune/logs/r4-1.log; identical shape in runs 2 and 3 at lines 86 and 85):
strata serve: prompt 65527 tokens = 0 reused + 65527 read in 12706 ms (5157.2 tok/s), 223 generated in 1651 ms (135.1 tok/s), drafts accepted 131 of 206, 1 checkpoints
strata serve: decode expert cache hit rate: 99.3% (143966 hits / 144957 lookups); 3 more read by the GPU over PCIe or from another GPU (0.0% of all 144960 routed)
strata serve: KV streaming: 95.89% of 1885524 block reads hit VRAM, 312.4 MiB read from RAM
strata serve: suffix drafts: 5 windows, 19 of 21 drafts accepted
iq_embed_rows: invalid argument
The failing request (the next tier, 131072 tokens) produced no `prompt ... read` line at all - the failure is at prompt ingestion, before any chunk progressed.
Analysis: `iq_embed_rows` launches one row per prompt token. Three sibling kernels guard the same CUDA grid-y limit of 65535 (src/kernels/cuda/qsa_select.cu:1168 `grid.y > 65535`; native_rope.cu:191,223 rows > 65535; sampler.cu:1196 tokens > 65535), but the iq_embed_rows path has no guard or row tiling - a 65536-row launch exceeds grid.y (max 65535). Consistent with this: a FIXED chunk of 61440 (below the limit) completes a full 9-tier x 3-run ladder on the same machine/config with zero occurrences of the error (prompt-path borrow 23.70 GiB vs 9988 slots at 65536, proportional).
Suggestion: add the guard + row tiling (or clamp) to the embedding upload path, like the sibling kernels.
Secondary: `--prefill auto:NN` silently clamps to 32768 for NN >= 32768 (src/program/generate.cpp:2041-2049); I first tried `auto:65536` expecting a larger chunk and got 32768 with no notice. Documenting (or reporting) the ceiling would help.
Happy to paste the full three run logs if useful.
Related on this site
Quick links to related pages.