The IQ3-XXS quantization was benchmarked on an RTX 3090 Ti with 24 GB VRAM. Inference speed: 2255.1 reading tps, 93.1 writing tps; power draw: 600 W. Measuring yourself is simple: record the read and write TPS in Strata's performance panel, note the total power reported by the system monitor, and compare against the published spec sheet.
Why iq3-xxs works on this GPU
The RTX 3090 Ti has a Tensor Core layout that maps well to the 24 GB of GDDR6 memory used by the iq3-xxs format. The 600 W power ceiling is reached during full model loading but settles lower under steady inference. Using IQ3-XXS keeps the model weights compact enough to stay inside the VRAM budget while still leaving headroom for the runtime context.
How to verify on your own machine
- Launch Strata and load the local model with the iq3-xxs quantization enabled.
- Open the profiling view and copy the read and write TPS numbers after a stable warm-up period.
- Check the system power meter (e.g., the OS power indicator or external wattmeter) while the engine runs a generation pass.
- Confirm that the total consumption stays close to the 600 W peak measured in the spec sheet.
The following table summarizes the key measured figures:
| Metric | Value |
|---|---|
| VRAM | 24 GB |
| Reading TPS | 2255.1 |
| Writing TPS | 93.1 |
| Power W | 600 |
IQ3-XXS is a low-bit quantization that shrinks model size without losing much accuracy. By discarding the least important weights, the model fits into 24 GB VRAM and runs fast on Tensor Cores. This makes the RTX 3090 Ti usable for local, private AI tasks that would otherwise need a datacenter GPU.
Practical implications
If you are considering Strata for on-device inference, the iq3-xxs profile shows you can expect near‑real‑time generation with modest power use. Keep an eye on thermal throttling if you push the card beyond idle temperatures; the 600 W ceiling reflects peak load, not sustained usage.
FAQ
[ { "q": "What does the IQ3-XXS label mean?", "a": "IQ3-XXS is a three‑bit quantization scheme with an extra optimization pass (XXS) that preserves more precision than standard 3‑bit formats. It reduces model size while keeping generation quality close to the original." }, { "q": "Can I run the model on a laptop?", "a": "The measured profile is based on a desktop RTX 3090 Ti. Smaller GPUs will have lower TPS and higher relative power overhead, so the numbers will differ. Always test on your own hardware first." }, { "q": "How much electricity does it use?", "a": "The engine peaks around 600 W when loading and loading the full model, then settles lower during steady inference. At typical usage that translates to several kWh per day depending on hours of active running." } ]