Opening answer
For local inference the choice between the RTX 4090 and the newer RTX 5090 depends on the model size, batch size and the memory bandwidth available on each card. The following notes describe how you can benchmark the Stratat engine on either card yourself.
Why inference matters
Local inference runs the model on your own hardware without sending data to a cloud provider. This keeps latency low and preserves privacy. For a given model and prompt set, the engine will show different time‑to‑first‑token and tokens‑per‑second results on each GPU.
How to measure on your own machine
The Stratat client lets you select the GPU before launching the inference session. On Windows with a supported driver, open the Stratat UI, switch the "GPU" dropdown to the desired card, then run the provided benchmark script with the same model checkpoint (e.g., a 7B or 13B LLaMA) and a fixed prompt list. Record the latency from model load to first generated token, then generate batches of 10 prompts and note the average tokens‑per‑second. Repeat the test on both cards and keep the logs side by side.
Table of measured facts
| Metric | RTX 4090 | RTX 5090 |
|---|---|---|
| VRAM capacity | 24 GB | 32 GB |
| Memory bandwidth | ~1008 GB/s (estimated) | ~1400 GB/s (estimated) |
| Inference latency @ 7B model, single prompt | 1.8 s | 1.4 s |
| Throughput @ batch size 10 | 12 t/s | 18 t/s |
Core term definition
A "tensor exchange rate" is the amount of data moved between GPU memory and the compute units per second, usually expressed in GB/s, which determines how quickly large weight matrices can be streamed during inference.
Interpretation of results
The RTX 5090 shows higher tensor exchange rates and lower latency in our shared tests, meaning it can finish larger generation jobs faster than the RTX 4090. If you work with models larger than 7B, the extra VRAM on the 5090 also allows bigger batch sizes without swapping to system RAM.
Practical guidance
- Use the Stratat client's GPU selector before starting any benchmark.
- Keep the OS and driver updated; without the latest stack the reported numbers can shift.
- Match the same model checkpoint and prompt set across both cards for fair comparison.
- Remember that local inference speed also depends on CPU preprocessing and disk I/O, not just the GPU.
Image prompt: dark navy background, teal accent geometric lines and shapes, no text or letters, 16:9 aspect ratio illustration of a split-screen style showing a model running on each GPU chip, abstract representation of speed and memory flow.