Summary

The RTX 4090 with the q2-0 quantization loads into 24 GB of VRAM and answers prompts at about 4000 reads per second while writing results at 109.3 writes per second under Stratat's local engine, drawing around 550 W of power.

Why q2-0 was chosen

The q2-0 format cuts the model size roughly by half compared with full‑precision releases, which fits inside the 24 GB VRAM limit of the 4090 and still leaves room for the runtime and OS buffers. This balance lets users run larger local models on consumer hardware without needing multi‑GPU setups.

How to verify on your own machine

Run Stratat's benchmark utility on a fresh boot and note that the model occupies 24 GB of VRAM, confirming the fit field. Measure the read and write TPS with the provided latency pings, then compare the power reading against the 550 W label during active inference to see the real consumption.

MetricValueNotes
vramGb24Fits inside RTX 4090 memory
readingTps4000Text generation throughput
writingTps109.3Response serialization throughput
powerWatts550Peak draw during load

The q2-0 quantization keeps the model compact enough for the 4090 while preserving usable quality for local chat and data tasks. Stratat exposes a simple UI for loading the model, launching inference, and reviewing latency stats on the same machine.

One core definition: quantization q2-0 is a reduced‑bit representation of a neural network that lowers precision to save memory and compute, enabling large models to run on GPUs with limited VRAM.

FAQ

  • What does q2-0 mean for me?

It means a smaller, lower‑bit version of the model that fits in the RTX 4090's 24 GB of VRAM and runs faster than higher‑precision versions. - How do I know my power usage is correct? Use the system's power meter alongside Stratat's telemetry; the readings should hover near the 550 W specification during inference. - Can I run other quantizations instead? Other quantizations exist, but they often exceed VRAM or drop performance; q2-0 offers the best balance for this hardware.