Issues / #3610

#3610 [Engine] It is not possible to set cuda_device in vision

open · @FranBarInstance · 2 comments · View on GitHub

Server & APINVIDIA / CUDA

Description

I have a second GPU with 4GB of VRAM, which I use for my monitors and applications; I also utilize it for vision-related tasks, thereby freeing up all the memory on the primary GPU for inference.

I am trying to migrate my configuration from version 1.45 to version 1.46, but I cannot find a way to use `cuda_device`

```
 "vision": {
  "exe": "...",
  "mmproj": "...",
  "model": "...",
  "gpu": true,
  "cuda_device": 1,
  "max_tokens": 1024
 }
```

I haven't been able to get it to work, and as a result, performance is significantly lower in the new version.

Strata version. 1.46.x

Related on this site

Quick links to related pages.