The macstudio_m5ultra_96-iq2-xs configuration delivers exceptional throughput for local inference, achieving a reading speed of 2284.8 tokens per second while maintaining a stable power draw of 360 watts. This setup utilizes the full 96 gigabytes of unified memory to run the iq2-xs quantization, ensuring that the model fits comfortably without requiring external storage swapping.

Why This Configuration Matters

Running large language models locally requires balancing memory capacity with processing speed. The 96 gigabytes of unified memory in the Mac Studio provides a massive pool for the model weights and context window. The iq2-xs quantization reduces the precision of the weights significantly, allowing a very large model to reside entirely within this 96 gigabyte limit. This balance is critical because it prevents the system from relying on slower disk I/O for context, which would drastically reduce the tokens per second.

How to Verify These Numbers

To verify these metrics on your own machine, you must ensure the Strata engine is running with the specific iq2-xs quantization file. You can check the reading speed by initiating a long context generation and measuring the output over time. The system reports a reading throughput of 2284.8 tokens per second and a writing speed of 64.8 tokens per second. To measure the power consumption accurately, monitor the system's energy meter during a sustained inference task. The data shows a consistent draw of 360 watts, which is essential for calculating your operational costs based on your local electricity rate.

Performance Metrics

The following table summarizes the core performance indicators for this specific hardware and software combination.

MetricValueSignificance
Memory Capacity96 GBAllows large models to fit without swapping.
Reading Speed2284.8 tpsHigh throughput for rapid text generation.
Power Draw360 WIndicates high performance energy cost.

Cost Calculation

Understanding the cost of running this setup helps you budget for long inference sessions. With a power draw of 360 watts, the system consumes 0.36 kilowatt-hours per hour. At a local electricity rate of $0.17 per kilowatt-hour, the cost to run this configuration for one hour is approximately $0.06. This makes it a viable option for developers who need high speed without the recurring costs of cloud API fees.

The iq2-xs quantization is a technique that reduces the precision of the model weights to 2-bit values, significantly decreasing memory usage while attempting to maintain acceptable accuracy.