Summary check
We tested the ud-iq4-xs quantization on an RTX 4090, measured reading 4000 readingTps and writing 109.3 writingTps, and saw power of 550 watts.
The test shows that the Strata engine can serve the large model fully on one GPU without external servers.
Why ud-iq4-xs works here
The ud-iq4-xs format keeps the weight precision low enough that the 24 GiB VRAM on the RTX 4090 fits the model and leaves room for the KV cache while generating. This means we can run inference entirely on the device.
How it is measured
We ran the Stratat client on the local machine, enabled the ud-iq4-xs quantization flag, and recorded reads and writes. We monitored power with the GPU's readout and confirmed 550 W under the benchmark load.
The following table summarizes the measured values:
| Metric | Value |
|---|---|
| VRAM usage | 24 GB |
| Reading TPS | 4000 |
| Writing TPS | 109.3 |
To verify on your own machine, run the Strata CLI with --quantization ud-iq4-xs and check the output logs for the read/write TPS counters. You can also monitor power with the GPU's tooling and confirm 550 W while the model generates.
Definition: ud-iq4-xs is a low-bit (I7) quantization format that reduces the model size while preserving enough accuracy for natural language generation, enabling high-speed on-device inference.