The RTX 5090 running the rtx5090-ud-iq4-xs quantization delivers exceptional throughput for local AI inference, achieving 5,803 tokens per second in reading and 168.4 tokens per second in writing. This configuration fits comfortably within the 32GB VRAM limit with a "Good" fit rating, making it a highly efficient choice for users who want to maximize output speed without exceeding memory constraints. The primary trade-off is the significant power consumption of 700 watts, which impacts your electricity costs but is justified by the raw performance gains.

Understanding Throughput and Latency

When you run this specific model configuration, the most noticeable metric is the reading speed. A reading speed of 5,803 tokens per second means that you will see text appear almost instantaneously. This is significantly faster than typical local setups, which often struggle to break the 100 tokens per second barrier. The writing speed, or generation speed, sits at 168.4 tokens per second. While slower than reading, this is still very fast for complex reasoning tasks. You can verify these numbers by running a simple benchmark script on your own machine and timing the output. The high throughput suggests that the RTX 5090 is effectively utilizing its memory bandwidth to feed the model.

Power Consumption and Operational Cost

The RTX 5090 draws 700 watts under load for this specific task. To calculate your actual cost, you need to look at your local electricity rate. Using the average US rate of $0.17 per kWh, running this setup for one hour will cost you approximately $0.119. While this seems small, running it for extended periods or multiple instances adds up. You can monitor your power draw using software like HWInfo to see the real-time wattage. It is important to ensure your power supply unit (PSU) can handle this peak load without throttling.

VRAM Efficiency and Model Fit

The model requires 32GB of VRAM, which matches the capacity of the RTX 5090 exactly. The "Good" fit rating indicates that the model fits well without excessive swapping or out-of-memory errors. This is a critical metric because it ensures stability during long generation tasks. If you try to load a larger model, you will likely hit the VRAM ceiling and experience severe performance degradation. You can check your VRAM usage in the task manager or via nvidia-smi to confirm the model is using the available memory efficiently.

MetricValueUnit
Reading Speed5803tokens/sec
Writing Speed168.4tokens/sec
Power Draw700watts
VRAM Usage32GB

The "fit" metric defines how well the model architecture aligns with the hardware memory constraints, ensuring stable and predictable performance without overflow errors or inefficient memory paging.

Practical Usage Considerations

For most users, this configuration is ideal for running large language models locally. The high reading speed ensures a smooth user experience, while the writing speed is sufficient for most creative and analytical tasks. The main consideration is the power draw. If you are running this on a laptop or a system with limited cooling, you may need to adjust settings. However, for a desktop setup with proper cooling, the RTX 5090 handles this workload with ease. You can test the stability by running a long context window and monitoring for any errors. The combination of speed and memory efficiency makes this a top-tier choice for local AI enthusiasts.