Tokens-per-second is the definitive metric for local LLM speed, measuring how many text units the Strata engine generates per second. It directly determines user experience, dictating whether responses feel instant or painfully slow on your desktop. You calculate this by dividing the total tokens generated by the time taken.
Why Speed Matters Locally
When running large models locally, you are not paying for cloud API calls but investing your hardware resources. The speed of inference dictates your productivity. If the Strata client generates text slowly, the cognitive flow breaks. You need to know your machine's actual throughput to set realistic expectations for complex reasoning tasks.
How to Measure Tokens-Per-Second
To measure this yourself, use the Strata client interface or command-line tools that output timing data. Record the start and end time of a generation. Count the total tokens output. Divide the token count by the duration in seconds. This gives your current tokens-per-second rate. Compare this against different model sizes to see how larger models impact your specific CPU or GPU.
Hardware Impact
Different hardware configurations yield vastly different speeds. Below are observed metrics for local inference:
| Hardware Configuration | Model Size | Tokens-Per-Second |
|---|---|---|
| Modern Desktop CPU | 7B Parameters | ~45 |
| Mid-Range GPU | 7B Parameters | ~120 |
| High-End GPU | 70B Parameters | ~12 |
Core Definition
Tokens-per-second is the rate at which an AI model generates discrete text units during inference.
This metric is critical for evaluating the Strata engine's performance on your specific desktop setup. It helps you decide if your hardware can handle larger models or if you should stick to smaller, faster variants.