The rtx3090ti-ud-iq4-xs configuration delivers a balanced inference experience on the RTX 3090 Ti, effectively utilizing its 24GB of VRAM to maintain high token throughput while managing thermal and power constraints. This setup is particularly suited for users who prioritize stable, continuous generation over raw maximum speed, as the system maintains a steady power draw of 600W to sustain performance without immediate thermal throttling. By understanding the specific metrics of this quantization, you can better anticipate the behavior of your local Strata engine during extended sessions.
Understanding the Hardware Constraints
The RTX 3090 Ti is a powerful card, but it is also known for its high power consumption. When running the ud-iq4-xs quantization, the system draws up to 600 watts under load. This is a critical factor for users with smaller power supplies or those concerned about electricity costs in the United States, where the average rate is approximately $0.17 per kWh. The 24GB of VRAM is fully utilized to hold the model weights and context, ensuring that the model fits well without swapping to system RAM, which would drastically reduce performance.
Measuring Your Local Performance
To verify these metrics on your own machine, you should monitor the Strata engine's output during a benchmark run. Observe the reading tokens per second (TPS) to gauge how quickly the model processes your prompt, and watch the writing TPS to see the generation speed. You can also use hardware monitoring tools like HWInfo to track the GPU power draw in real-time. If your system consistently hits 600W, ensure your cooling solution is adequate to prevent throttling, which would lower the writing TPS below the expected 93.1 tokens per second.
Performance Breakdown
The following table summarizes the key performance indicators for this specific configuration:
| Metric | Value | Implication |
|---|---|---|
| VRAM Usage | 24 GB | Full utilization of GPU memory for model fit |
| Reading Speed | 2255.1 TPS | Fast context processing and prompt analysis |
| Writing Speed | 93.1 TPS | Moderate generation speed for coherent output |
Evaluating the Fit
The fit is described as Good, indicating that the model size and the quantization method align well with the hardware capabilities. This means you are getting the best possible performance for the resources available. While the writing speed is not the highest possible, it is sufficient for most interactive tasks. The high reading speed ensures that long context windows are handled efficiently, allowing for complex queries to be processed quickly.
Quantization is the process of reducing the precision of the model's weights to save memory and speed up computation, which is essential for running large language models locally on consumer hardware.