The RTX 3090 handles the rtx3090-ud-iq4-xs quantization effectively, delivering a strong balance of speed and memory efficiency within its 24GB VRAM limit. This configuration fits well on consumer hardware, allowing for rapid context processing without exceeding the physical constraints of the GPU.
Why This Configuration Works
The primary advantage of this setup is the "Good" fit rating, which indicates that the model weights and activation states occupy the available 24GB of VRAM without spilling over into system RAM. When running local AI models, spilling into system RAM causes a drastic reduction in speed because PCIe bandwidth is significantly slower than GPU memory bandwidth. By staying within the 24GB limit, the GPU can maintain high throughput. The ud-iq4-xs quantization reduces the precision of the model weights, allowing more parameters to fit into the limited space compared to higher precision formats like Q8_0 or even Q4_K_M in some cases.
How to Verify Performance
You can verify these metrics by running the Strata engine with the specific model file and monitoring the output tokens per second. The measured reading speed of 2094 tokens per second suggests that the model is not I/O bound during the context loading phase. However, the writing speed of 86.5 tokens per second is a more realistic indicator of interactive response time. You should monitor your power consumption using tools like HWInfo or the Strata dashboard to ensure the GPU does not exceed its thermal design power. At 500 watts, this is a power-intensive workload that requires a robust power supply unit, typically 850W or higher for the entire system.
Measured Specifications
| Metric | Value | Implication |
|---|---|---|
| VRAM Usage | 24 GB | Fits entirely in GPU memory for maximum speed. |
| Reading Speed | 2094 TPS | Fast context loading and retrieval. |
| Writing Speed | 86.5 TPS | Moderate token generation for chat interactions. |
| Power Draw | 500 Watts | High energy cost; requires strong PSU. |
Cost of Operation
Running this configuration costs money in electricity. At a national average of $0.17 per kWh, a one-hour session consumes 0.5 kWh and costs approximately $0.085. While this is not expensive per hour, it adds up during long training or fine-tuning sessions.
The term "quantization" refers to the process of reducing the precision of the numbers used to represent the model's weights, which saves memory and computation at the cost of some accuracy.