The RTX 5080 handles the iq2-xs quantization with excellent VRAM utilization, though users must manage high power draw. This configuration fits comfortably within the 16GB limit, delivering strong token generation speeds at the cost of significant energy consumption.
Understanding VRAM and Fit
When running large language models locally, the first constraint is always memory. The RTX 5080 provides 16GB of VRAM, which is the primary resource for loading the model weights and maintaining the context window. For the iq2-xs quantization, the fit is classified as good. This means the model loads entirely into the GPU without spilling over to system RAM, which would drastically reduce performance. You can verify this on your own machine by monitoring the VRAM usage in your system monitor or using GPU diagnostic tools during inference.
Measuring Speed and Throughput
Speed is measured in tokens per second (tps). In this specific setup, the reading speed (context processing) is 3,712.5 tps, while the writing speed (token generation) is 80.7 tps. These numbers indicate that the GPU is processing the input prompt very quickly but generating the response at a moderate pace. To check these numbers yourself, you can use command-line interface tools or the Stratat client to log the tokens generated per second during a test run.
Power Consumption and Cost
The system draws 510 watts under load. This is a critical factor for home lab enthusiasts. Using the average US electricity rate of $0.17 per kWh, you can calculate your cost. Running this setup for one hour consumes 0.51 kWh. Multiplying this by the rate gives a cost of approximately $0.0867 per hour. While not expensive per session, the high wattage means your GPU will run hot and consume more power than a less aggressive quantization might.
Summary of Metrics
| Metric | Value | Unit |
|---|---|---|
| VRAM | 16 | GB |
| Reading Speed | 3,712.5 | tps |
| Writing Speed | 80.7 | tps |
| Power Draw | 510 | Watts |
| VRAM Fit | Good | Status |
The iq2-xs quantization is a specific method of reducing model precision to save memory, allowing larger models to run on consumer hardware by sacrificing some numerical accuracy for significant space savings.