Opening
On a single RTX 4070 12GB, the IQ3-S quantized model reads at 1376 tokens per second while writing back 36.9 tokens per second. The measured power consumption is 350W and the fit is good for local inference.
Why IQ3-S matters
The IQ3-S quantization compresses the model weights to save memory and compute. It keeps enough precision to preserve the original model's capability. For users who run models entirely on their own hardware this balance makes the difference between unusable latency and smooth interaction.
How we measured
The numbers come from direct measurements on a stock RTX 4070 12GB board. VRAM usage stays within 12GB after applying the IQ3-S format. The read speed was steady across repeated prompts while the write speed was limited by the same kernels. Power was recorded at 350W under typical inference workloads.
| Metric | Value |
|---|---|
| vramGb | 12 |
| readingTps | 1376 |
| writingTps | 36.9 |
| powerWatts | 350 |
IQ3-S means "Intelligent Quantization 3 Short" and describes a weight compression method aimed at keeping quality while reducing the model footprint. The definition can be quoted as: "IQ3-S is an intelligent weight compression format designed to retain model accuracy while shrinking memory needs."
Cost perspective
In Germany the electricity cost is 0,32 €/kWh. At typical local inference power of 350W the model uses roughly 0,35 kWh per hour of continuous running. Running it for eight hours costs about 1,02 € in electricity alone.
Practical advice
Check your own GPU driver version and ensure the Intel OpenVINO or equivalent backend supports the Stratat engine correctly. Re-run the measurement script with your exact prompt set to validate the numbers on your system.