#486 [2x RTX 2080 Ti ] 262K context can OOM the weight arena / hard-freeze host; 204800 is stable

本文

# [Linux][2x RTX 2080 Ti] 262K context can OOM the weight arena / hard-freeze host; 204800 is stable

## Summary

On a Linux system with **2× RTX 2080 Ti 22 GB** using a layer split, a 262144-token context can exhaust VRAM during startup / allocation and, in the worst case, make the whole machine stop responding. Reducing the context to **204800** makes the same setup stable.

The concrete allocation failure I observed was:

```text
cudaMalloc(1538035200) weight arena failed
```

This was with **IQ3_XXS + vision enabled**. Because the failure can hard-freeze the host, I have not deliberately reproduced it again on 0.1.35 yet. I checked the 0.1.35 changes; #448 improves prompt-chunk diagnostics for split GPUs, but it does not appear to change this allocation path.

## Environment

- OS: Ubuntu 24.04
- RAM: 126 GB
- CPU: AVX-2, no AVX-512
- GPU: 2× NVIDIA GeForce RTX 2080 Ti 22 GB
- Interconnect: NVLink (NV2)
- GPU topology: both GPUs on CPU affinity 0-23
- Model: Swift 1.5 / Qwen3.8 Flash-Next GSQ-RCO **IQ3_XXS**
- Vision: enabled
- Layer split: auto, CUDA0 then CUDA1
- KV: int8
- KV resident: 32768
- Speculation: 4
- PCIe fraction: 0.35
- Spec min p: 0.70
- Pool workers: 9

The machine normally runs near full GPU utilization during generation.

## Reproduction shape

Problematic configuration:

```text
--context 262144
--kv int8
--kv-resident 32768
--spec 4
--spec-min-p 0.70
--pcie-frac 0.35
--pool-workers 9
--vision
--layer-split auto
```

With IQ3_XXS, this reached a point where a ~1.43 GiB device allocation failed:

```text
cudaMalloc(1538035200) weight arena failed
```

On at least one run, the system became effectively unresponsive rather than failing cleanly.

## Stable comparison

The same machine is stable with:

```text
--context 204800
--prefill 4096
--reserve 1800
--kv int8
--kv-resident 32768
--spec 4
--spec-min-p 0.70
--pcie-frac 0.35
--pool-workers 9
```

So the practical boundary on this machine is somewhere between ~205K and 262K with this model / vision / split configuration.

## Why this seems worth guarding

At 262K the combined VRAM consumers include:

- dense/model weights
- expert cache
- streamed/resident KV
- prefill buffers
- MTP/speculative buffers
- vision allocations
- CUDA workspace / weight arena

The engine can apparently commit enough of these earlier allocations that a later large weight-arena allocation fails. On this hardware the failure can be severe enough to make the host unusable.

## Request

Could Strata do one or more of the following?

1. **Budget the late weight-arena / workspace allocation before committing the expert cache / other large VRAM users**, especially with vision + layer split.
2. Keep a larger automatic safety margin when `--max-context` is very large.
3. If the requested context cannot fit safely, **fail before loading / allocating everything**, with a message suggesting a lower context such as 204800.
4. If possible, make allocation failure recover cleanly instead of leaving the GPU / host in a state that can hard-freeze Linux.
5. Log a compact per-GPU VRAM budget before the risky allocation (weights, expert cache, KV resident, prefill/workspace, vision, reserve), which would make this much easier to diagnose.

I can test a patch or provide a full startup log if there is a specific tracing flag you want enabled.

## Note about 0.1.35

I saw the new #448 diagnostics around prompt-chunk borrowing and split-card cache constraints. My GPUs are symmetric 22 GB cards, and this failure is a later `cudaMalloc(... weight arena ...)` OOM rather than the small-secondary-GPU prompt-chunk case, so I don't think #448 addresses this particular failure.