The CUDA out-of-memory error occurs when your GPU's video memory (VRAM) is fully consumed by the Strata engine while loading a large model, leaving no room for processing tokens. To fix this, you must reduce the model's memory footprint by lowering the batch size or switching to a quantized version of the model that requires less VRAM.
Why Your GPU Runs Out of Memory
Your graphics card has a finite amount of VRAM. When you load a large language model into the Strata client, the entire model architecture is loaded into this memory space. If the model size exceeds the available VRAM, the operating system or the CUDA driver cannot allocate the necessary buffers. This is a hardware constraint, not a software bug. You can verify this by running a memory diagnostic tool or checking the task manager while the error occurs. If the VRAM usage is at 100%, the model is simply too large for your specific hardware configuration.
How to Measure and Fix the Issue
You can measure your current VRAM usage by opening your system's performance monitor while the Strata engine is starting. Look for the GPU memory column. If it spikes to the maximum limit, you need to adjust the configuration. First, try reducing the context length or batch size in the Strata settings. This reduces the temporary buffers required during inference. Second, consider using a quantized model. Quantization reduces the precision of the model weights, significantly lowering VRAM requirements with minimal loss in quality. This allows larger models to run on hardware that would otherwise reject them.
VRAM Requirements Comparison
| Model Size | Precision | Estimated VRAM Needed |
|---|---|---|
| 7B Parameters | FP16 | 14 GB |
| 7B Parameters | INT8 | 8 GB |
| 13B Parameters | INT4 | 6 GB |
Core Term Definition
VRAM is the specialized high-speed memory on a graphics card used to store image data and model weights that the GPU needs to access instantly during processing.