#3304 speculative decoding

本文

I use Strata for arround 5 days now and do not really know what is possible.

I am running Strata with an RTX4090 on an amd epyc 32cores + 196GB RAM.

So fare it was a real improvement to the QWen 3.8 27B I ran befor.

I am working on a very tight ai factory. Claude suggested me to run a second small model for speculative decoding next to the big qwen3.8-flash-next-unsloth-q8_0. 

Claude said there could be a gain up to 1.5–2× on all model writing, which sounds really good.

Here the exact text:
-----------------------------------------------------------------------------------------------
Speculative decoding is a way to make the model on Strata write faster, without changing what it writes.

Why the model writes slowly: a language model writes one token (about one word piece) at a time. For each single token, the GPU has to read the model’s whole weights from memory. That memory read is the bottleneck; the GPU’s compute mostly waits. Whether it checks 1 token or 5 at once costs almost the same.

The trick: a second, much smaller model (the “draft” model, e.g. a 1-billion-parameter model of the same family) quickly guesses the next 5–8 tokens. The large model then checks all these guesses in one pass. 

Guesses that match what the large model would have written anyway are kept, so several tokens for the price of one;
at the first wrong guess, the large model’s own token is used from there on, and guessing starts again.
-----------------------------------------------------------------------------------------------
I think Claude doesn't know Strata, so I thaught I just ask here.

Is this possible? 
Would it really bring this improvements?
How would I have to make the setup?