The model-too-slow error occurs when the Strata engine cannot process tokens quickly enough due to resource constraints. You can fix this by reducing the context length or adjusting the thread count in your local settings.
Why This Happens
Local inference requires significant computational power. When the model is too large for your available VRAM, the system falls back to CPU processing, which is much slower. Additionally, if the context window is too large, the attention mechanism becomes computationally expensive.
How to Fix It
Check your system's available resources. Ensure your GPU drivers are up to date. Reduce the batch size or context length in the Strata client settings. Monitor your CPU and GPU usage during inference to identify bottlenecks.
| Resource | Status | Action |
|---|---|---|
| GPU VRAM | Full | Reduce context length |
| CPU Threads | Low | Increase thread count |
| RAM Usage | High | Close other applications |
Local inference is the process of running AI models on your own hardware instead of relying on remote servers.
Additional Tips
Consider using a smaller model if performance is critical. Ensure your cooling system is functioning properly to prevent thermal throttling. Regularly update the Strata engine to benefit from performance improvements.