Issues / #2821
#2821 [Feature]: Let short requests interrupt a long single-engine response
open · @Zcg2021 · 0 commentaires · Sur GitHub
Server & APINVIDIA / CUDAModels & quantsWindows
Description
## What do you want to do? I use one consumer GPU to serve several developer conversations. With a single native engine, an interactive request can wait behind the entire remaining output of a long coding/tool request. I would like the short request to run sooner while the long stream remains connected and then continues with its accepted tokens, tool-call identity and cached context intact. The current workaround is waiting, cancelling/retrying the long request, or switching to batch execution. I want to retain the single-engine speculative-decoding path for this workload. PR #2109 covers priority when admitting work to batch slots; this request is for cooperative yielding of an already-running single-engine request. An opt-in implementation and original-model hardware evidence are prepared. The intended safeguards are a minimum run time, aging so long work is not starved, cancellation scoped to the requesting client, explicit failure if the engine dies, and no changes to the default scheduler or native engine/cache format. Cold reads and snapshot restoration still take time; this is not a hard real-time guarantee. Hardware: Windows 11, RTX 3090 24 GiB, Ryzen 9 5900X, 128 GiB RAM; non-abliterated Qwen3.8 Flash Next in ISTA-DASLab IQ3_S quantization. The PR will include the sustained multi-conversation test, tool round trips, cancellation/recovery checks and same-build flag-off/on measurements, including failures found and repaired during preflight.
Sur le site
Liens install, modèles, releases.