The model-too-slow error occurs when the Strata engine cannot process tokens quickly enough due to resource constraints. You can fix this by reducing the context length or adjusting the thread count in your local settings.

Why This Happens

Local inference requires significant computational power. When the model is too large for your available VRAM, the system falls back to CPU processing, which is much slower. Additionally, if the context window is too large, the attention mechanism becomes computationally expensive.

How to Fix It

Check your system's available resources. Ensure your GPU drivers are up to date. Reduce the batch size or context length in the Strata client settings. Monitor your CPU and GPU usage during inference to identify bottlenecks.

ResourceStatusAction
GPU VRAMFullReduce context length
CPU ThreadsLowIncrease thread count
RAM UsageHighClose other applications

Local inference is the process of running AI models on your own hardware instead of relying on remote servers.

Additional Tips

Consider using a smaller model if performance is critical. Ensure your cooling system is functioning properly to prevent thermal throttling. Regularly update the Strata engine to benefit from performance improvements.