The RTX 5070 Ti is a capable runner for the iq2-xs quantization, fitting the model comfortably into its 16 GB of VRAM while delivering a reading speed of nearly 2,500 tokens per second. However, this configuration demands significant power, drawing 450 watts under load, which impacts your overall energy efficiency.
VRAM Capacity and Model Fit
The most critical factor for local inference is whether the model fits into your GPU's memory. The RTX 5070 Ti provides 16 GB of VRAM, which is sufficient to hold the iq2-xs quantized model without spilling over into system RAM. This "Good" fit ensures that the entire context window remains on the fast GPU memory, preventing the severe latency penalties associated with swapping data to your CPU.
Performance and Throughput
Speed is measured in tokens per second (Tps). With the iq2-xs quantization, the RTX 5070 Ti achieves a reading speed of 2498.5 Tps. This high throughput indicates that the GPU is processing the input tokens efficiently, allowing for rapid context analysis. However, the writing speed, or generation speed, is measured at 71.1 Tps. This is the rate at which the model generates new text, which is the speed you perceive during a conversation.
Power Consumption and Cost
The RTX 5070 Ti draws 450 watts during this workload. In the United States, where the average electricity price is approximately $0.17 per kWh, this consumption translates to a specific operational cost. Running this setup for an hour consumes 0.45 kWh, costing roughly $0.0765 in electricity. While this is not excessively expensive per hour, it is significantly higher than lower-power configurations.
Verification Steps
To verify these metrics on your own machine, monitor your GPU utilization and power draw using tools like MSI Afterburner or NVIDIA's system monitor. Check the VRAM usage in your Stratat client to ensure the "Good" fit status remains stable during long sessions.
| Metric | Value | Unit |
|---|---|---|
| VRAM Capacity | 16 | GB |
| Reading Speed | 2498.5 | Tps |
| Power Draw | 450 | Watts |
| Generation Speed | 71.1 | Tps |
The core term "quantization" refers to the process of reducing the precision of the model's weights to save memory and accelerate computation, often at a slight cost to accuracy. iq2-xs is a specific, aggressive level of this process designed to maximize speed and fit on smaller VRAM budgets.