#5767 [Engine] spec-prob verify: alloc-failure fallback to plain sampling is silent (no log) on the production path
open · @nevermore131315 · 11 评论 · 去 GitHub 看
NVIDIA / CUDAModels & quantsWindowsLinux
说明
### Area
Engine
### Desktop app, Terminal or source?
Source — static analysis of the engine source; no app or CLI session involved.
### What happened
**Correction of the original report:** the headline claim — "silently and permanently disables speculative decoding for the process lifetime" — was wrong. Re-reading the gate, the lazy `d_spec_` allocation sits **inside the per-window gate** in `verify()` (`if (sampled && (measure || (spec_q_ != nullptr && spec_nq_ > 0 && T > 1)) && T <= kVerifyMaxT)`), so the `d_spec_ == nullptr` check re-attempts the `cudaMalloc` on every verify window while the spec-prob gate is active. A transient `cudaMalloc` failure therefore degrades only the windows during which the allocation keeps failing, and recovers on the first window whose allocation succeeds — the behavior originally requested as a fix ("re-attempt the allocation on later windows") is already what the code does.
The surviving, much narrower issue: on the non-measure (production) path, a window whose `d_spec_` allocation or copy fails falls through to plain `sample_tokens()` with **no log line** — under sustained VRAM pressure, spec-prob decoding can run degraded indefinitely with zero diagnostics. The `measure` path does surface it (`err = "verify: proposal sampler unavailable or allocation/copy/launch failed"`), but production absorbs the failure silently.
Suggested fix: log a one-time stderr warning when a production-path window falls back because the proposal-sampler allocation/copy failed, consistent with the engine's other self-diagnostic lines.
### Reproduction status
Unconfirmed static analysis — code reading only; no runtime failure was observed and nothing was executed. The original report's "permanent, no retry" characterization was incorrect and is corrected above; the silent-fallback observation stands.
### Strata version
0.2.11 (engine 0.1.57), read directly from the v0.2.11 tag (`src/core/verify.cpp` ~lines 3598–3627). The identical code is present in the current dev tree.
### Graphics card, memory and system
Static analysis only — no runtime involved. Source read on Linux (Fedora 45). Context: audited for a 3-GPU NVIDIA CUDA deployment (3090 + 3080 + 4070 Ti Super), but the finding is hardware-independent; it only affects `STRATA_SPEC_PROB` deployments or measure builds, since a plain exact-match config never enters this block.
本站相关内容
相关页面的快捷入口。