Pull requests / #2131

#2131 serve: the prompt path's loan has room for a short request that keeps 0.1.39's ring (#2090)

open · @ischencheng · 0 comentários · No GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

Fixes #2090.

With `--prefill auto` the serve scan sizes the prompt path's loan for the chunk it picked, with the byte-budget ring. A request whose chunk is at most 0.1.39's pick (`small` in `set_ring_budget(..., small)`) reads with 0.1.39's ring instead. On a pack with bigger blobs than Q2_0's that ring has more slots (384 against 227 on IQ3_XXS), so the short request can need more of the cache than the loan holds. `lend` clamps it to the loan's first slot, `relayout` runs out of room, and the engine exits with "lending the prompt path its slots failed". That is why it only showed with `--prefill auto` and only at some cache sizes in the report.

The loan now has room for both: `max(part_slots(chunk), part_slots(small))`. The same helper is used where the loan is sized again (the step-down after a failed init, and `relend` under `--vram-elastic`). `lend` is unchanged, so each request still lends only the slots its own chunk needs. A request that fitted before lends the same slots. Where 0.1.39's chunk needs no more slots than the chosen one, the loan does not change at all.

Measured on Modal, one L4 24 GB (sm_89, Linux), the official IQ3_XXS pack, upstream main 7695859, `--prefill auto --max-context 131072 --kv int8`. I set `--expert-cache N` by hand to land on the cache sizes where this happens. Each start gets four fresh prompts whose first part rounds up to a 2048 / 3072 / 4096 / 6144-token chunk.

| `--expert-cache` | auto chunk, ring | the short chunk needs | loan main -> this PR | main | this PR |
|---|---|---|---|---|---|
| 905, 910, 915 | 2816, 227 slots | 2048: 1059-1060 slots | 1023-1024 -> 1059-1060 | engine exits, 3 of 3 starts | 0 of 3 |
| 1088, 1094, 1100 | 3840, 227 | 3072: 1288-1289 | 1252-1254 -> 1288-1289 | 3 of 3 | 0 of 3 |
| 1284, 1290 | 4864, 227 | 4096: 1517 | 1481-1482 -> 1517 | 2 of 2 | 0 of 2 |
| 900, 1130, 1296 | 2816 / 4096 / 5120, 227 | fits | 1024 / 1309 / 1539, unchanged | 0 | 0 |

Two more runs of some of the failing rows gave the same result (main 5 of 5 starts, this PR 0 of 5). Main's error is the one in the report, with 13 GB of VRAM free:

```
lending the prompt path its slots failed: prefill: device buffers for a chunk of 2048 tokens do not fit (13033 of 22563 MiB free at the failure; 1719 MiB taken, the failing buffer wanted 20 MiB)
```

Replies (temperature 0, thinking off, 64 tokens) to the 2048 / 3072 / 4096 prompts are the same on main and this PR at every cache size where main answered them. The 6144 one changes between starts of main itself on this setup (3 different replies in 4 starts at `--expert-cache 900`); this PR's reply was one of main's.

`src/program/generate.cpp` also builds for HIP: compiled it in a ROCm 7.0 build for gfx1201 (that one file, no GPU), with the same 98 warnings as main.

Not tested: Windows, the reporter's IQ3_S pack and 4070 Ti, a HIP run, layer splits, `--vram-elastic`. `sycl/src/program/generate.cpp` has the same scan and is not changed here. The resident RAM mode sizes its RAM copy's lend region from `plan_lend`, which still counts the chosen chunk only, so there a short request's extra slots are read from the model file during that prompt.

No site

Links install, modelos, releases.