The RTX 3090 paired with the iq3-s quantization delivers a high-throughput local inference environment, generating tokens at 86.5 per second while consuming 500 watts of power. This configuration achieves a good fit within the card's 24GB VRAM, allowing for robust local model execution without external dependencies.
Why iq3-s Fits the RTX 3090
The iq3-s quantization is specifically designed to balance model fidelity with memory efficiency. By utilizing the RTX 3090's 24GB of VRAM, this setup ensures that the model fits comfortably, which is critical for maintaining stability during long inference sessions. The "Good" fit rating indicates that the model does not spill over into system RAM, preserving the high bandwidth of the GPU memory for faster processing.
Measuring Throughput and Power
To verify these metrics on your own machine, you must monitor the hardware during active inference. The reading speed of 2094 tokens per second suggests that the system can load context windows rapidly. However, the writing speed is limited to 86.5 tokens per second, which is the actual output rate you will experience. Power consumption is a significant factor; at 500 watts, the card draws substantial electricity. Using a local price of $0.17 per kWh, you can calculate the exact cost per hour of operation to determine if this setup is economically viable for your specific use case.
Key Performance Metrics
The following table summarizes the core performance indicators derived from direct measurement.
| Metric | Value | Context |
|---|---|---|
| VRAM Usage | 24 GB | Total available memory for the model |
| Reading Speed | 2094 tps | Context loading capacity |
| Writing Speed | 86.5 tps | Actual token generation rate |
| Power Draw | 500 W | Energy consumption under load |
Understanding Quantization
IQ3-S is a quantization format that reduces the precision of model weights to save memory while attempting to maintain acceptable quality.
The trade-off between speed and power is central to this configuration. While the reading speed is high, the writing speed defines the user experience. Users should ensure their power supply unit (PSU) can handle sustained 500-watt loads, especially during peak inference periods.