#6613 [TUI] Inkling-2-Mini SQ-15 planned as full-VRAM on a 16 GB Radeon runs out of memory at about 8.6K tokens of a 32K context

open · @Vyeche · 0 comentários · No GitHub

BenchmarksServer & APIAMD / HIPModels & quantsLinux

Descrição

### Area

[TUI]  (the Terminal: strata launch, connect, monitor)

### Desktop app, Terminal or source?

Terminal (the `strata` command)

### What happened

On a 16 GB Radeon, `strata launch` plans Inkling-2-Mini SQ-15 at 32K context as fully in VRAM (`--vram-experts-gb 12.28 --ram-experts-gb 0.00`). The model loads and answers short prompts, then a fresh prompt of roughly 8.6K tokens ends the engine with `fatal GPU error: [ink-hip] kv grow v: out of memory` (HTTP 503, the engine restarts). So the plan promises 32K and the card holds about 6.5K.

Graphics memory, read from `mem_info_vram_used` (card total 16,304 MiB; 359 MiB in use with no model loaded, the desktop runs on this card). Each request is a new one-message conversation, `max_tokens` 8:

| Point | VRAM in use |
|---|---:|
| No model loaded | 359 MiB |
| SQ-15 loaded, before any request | 15,375 MiB |
| After a 734-token prompt | 16,076 MiB |
| After a 2,171-token prompt | 16,185 MiB |
| After a 4,328-token prompt | 16,218 MiB |
| After a 6,484-token prompt | 16,226 MiB |
| Next prompt (about 8.6K tokens) | `kv grow v: out of memory`, 503 |

Against the plan in `strata_control/inkling_fit.py` for this card (`compute("SQ-15", 15.92, …, ctx=32768)` returns `vram_gb 12.28, full_vram True`): dense 2.07 GB + experts 12.28 GB + driver reserve 0.85 GB + KV for 32K 1.25 GB = 16.45 GB of 17.09 GB. Measured: loading alone takes 15,016 MiB (15.75 GB) over the idle state, which is 1.4 GB more than the 14.35 GB file, and the first 734-token prompt adds another 701 MiB. That leaves 78 MiB at 6.5K tokens.

Two more data points with experts moved to RAM by hand:

| `--vram-experts-gb` / `--ram-experts-gb` | 9,783-token prompt | Decode |
|---|---|---:|
| 12.28 / 0.00 (the plan) | out of memory | 69 to 88 tok/s on short prompts |
| 11.46 / 0.82 | out of memory (15,595 MiB in use after load) | 69 to 89 tok/s on short prompts |
| 10.50 / 1.78 | completes (14,390 MiB after load, 15,890 MiB after the test) | 23 to 34 tok/s |

What I expected: a plan that still has room for the context it was started with, or a recommendation of SQ-10 for a 16 GB card at 32K. SQ-10 on the same card: a 9,783-token prompt completes (not tested beyond that).

Unconfirmed, from reading the code only: `grow_pool` in `ink_hip_backend.hip.cpp` doubles the pool and keeps the old and the new buffer alive during the copy, so a grow needs about three times the old size for a moment; and the plan uses the card's total memory, not what is free.

### Smallest steps or command that reproduce it

```sh
strata launch --family inkling-2-mini --model sq-15 --context 32k --yes --no-tui --port 8081
# then one request with a fresh prompt of about 9K tokens
```

The first failure above was on the shipped 0.1.65 `hip-rx9070` package with a 9,783-token prompt. The table of VRAM readings was taken with PR #6594 applied (source build), because the shipped package returns NaN for every prompt of 64 tokens or more on this card (#6601).

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.19 · strata-inkling 0.1.65 (hip-rx9070 package, and a source build with PR #6594)

### Graphics card, memory and system

Radeon RX 9070 XT 16 GB (gfx1201), 92 GB RAM, Ryzen 9 9950X3D, Pop!_OS, Linux 7.2

### Model, size and settings

Inkling-2-Mini SQ-15, 32K context, pictures on (CPU reader), DSpark drafting (default), 1 slot

### Logs

```
[strata] native ERR fatal GPU error: [ink-hip] kv grow v: out of memory
[strata] the GPU reported a fatal error ([ink-hip] kv grow v: out of memory); the engine ended (exit code None) and is started again; the request that was running failed and can be sent again
```

This report was prepared with the help of an AI assistant; the measurements were run on the hardware above.

No site

Links install, modelos, releases.