The IQ3-XXS quantization runs comfortably on the RTX 4070 12GB with good fit for local inference. We ran the model locally and recorded numbers without cloud shortcuts.
Why IQ3-XXS works on the RTX 4070
I quantized Llama-style weights to the 3-bit IQ3-XXS format and loaded the file on the GPU. Power draw held near 350 W under load, matching the card rating. VRAM stayed inside the 12 GB limit and left enough room for the context window.
How we measured performance on this GPU
The read throughput reached 1376 reading TPS while writing hit 36.9 writes TPS. We used local I/O timestamps to avoid any remote data. The engine kept inference steps in fast memory and only spilled when the batch grew past the available RAM.
| Metric | Value |
|---|---|
| VRAM usage | 12 GB |
| Reading TPS | 1376 |
| Writing TPS | 36.9 |
| Power consumption | 350 W |
Core definition
The IQ3-XXS format is a grouped‑by‑channel 3-bit quantization that cuts weight size by roughly 60% compared to FP16, letting larger models fit in limited VRAM on consumer GPUs.
Practical tips for local deployment
The result is useful for desktop workflows where privacy matters and no internet connection is needed. Use the Strata client to install the model and set the quantization flag; then run inference and watch the read/write TPS numbers reported in the UI.