The rtxpro6000-ud-iq4-xs quantization configuration delivers a highly efficient balance for local inference on the RTX PRO 6000 Blackwell, offering a good fit within the 96GB VRAM constraint while maintaining respectable throughput speeds.

Hardware Context and VRAM Management

The RTX PRO 6000 Blackwell provides a substantial 96GB of VRAM, which is the primary resource for running large language models locally. The ud-iq4-xs quantization is selected because it fits comfortably within this memory limit, classified as a Good fit. This means the model can load entirely into the GPU memory without offloading to the slower system RAM, which is critical for maintaining low latency during generation.

Throughput and Power Analysis

Performance is measured in tokens per second (Tps). The reading speed of 8987.9 Tps indicates rapid context processing, allowing the system to ingest large documents or conversation histories quickly. However, the writing speed of 171.1 Tps represents the actual generation rate for the user. At the local electricity price of $0.17 per kWh, running this setup at a peak power draw of 720 watts incurs a specific operational cost. Users can calculate their own hourly expenses by multiplying 0.72 kilowatts by $0.17 to understand the financial impact of prolonged inference sessions.

Benchmark Data

MetricValueUnit
VRAM96GB
Reading Speed8987.9Tps
Writing Speed171.1Tps
Power Consumption720Watts

The ud-iq4-xs quantization is a method that compresses the model weights to 4-bit precision while preserving specific architectural details to minimize the loss of reasoning capability compared to standard 4-bit models.