Running the iq3-xxs quantization on an RTX 3090 delivers a compelling balance of speed and memory efficiency, though it demands significant power. The model fits comfortably within the 24GB VRAM limit, enabling a high throughput of 2094 tokens per second for reading, making it a viable option for local inference despite the steep electricity costs.

Performance and Power Dynamics

The RTX 3090 is a powerful consumer card, but it is not energy-efficient. When running the iq3-xxs quantization, the GPU draws approximately 500 watts. This power consumption is a critical factor for users calculating their operational costs. At the local average electricity rate of $0.17 per kWh, running this setup continuously is expensive. However, the performance metrics justify the cost for users who need rapid local generation without cloud latency.

Verification Steps

You can verify these metrics on your own machine by monitoring the GPU utilization and power draw during a sustained inference task. Use tools like `nvidia-smi` or hardware monitoring software to observe the wattage. The tokens per second (TPS) can be measured by timing the generation of a standard prompt. Ensure the model fits in VRAM by checking for out-of-memory errors, which the fit score suggests will not occur with this configuration.

Measured Specifications

The following table summarizes the key performance indicators for this specific hardware and software combination.

MetricValueNote
VRAM Capacity24 GBSufficient for iq3-xxs model
Reading Speed2094 TPSHigh throughput for local use
Power Draw500 WattsRequires robust cooling
Cost per kWh$0.17US average estimate

Understanding Quantization

Quantization is the process of reducing the precision of the model's weights to save memory and speed up computation, which allows complex models to run on consumer hardware.

By choosing iq3-xxs, you are prioritizing speed and memory fit over maximum accuracy. The RTX 3090 handles this load well, but the 500W draw means you must consider your power budget. The 2094 TPS reading speed is impressive, allowing for near-real-time interaction. This makes the setup suitable for chat applications where latency matters more than perfect statistical accuracy.