Issues / #2855

#2855 setup: vision encoder not rebuilt for a new GPU arch (sm_61) when the engine is, then fails with an empty error

open · @renaudrenaud · 0 comentários · No GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsLinux

Descrição

**What happened.** An install built for a Tesla V100 (engine and `strata-vision` for sm_70, Docker image based on `Dockerfile.v100`) was started on two Tesla P40s (sm_61) with `GPUS=0,1`. Setup noticed the new architecture and recompiled the engine:

```
Compiling the engine for sm_61, sm_70 (a card it had no code for; 10-20 minutes, once) ...
[ok] engine compiled: /opt/strata/engine-cuda12/strata
```

but it did not rebuild the image encoder. The server then stopped at start:

```
loading the vision encoder ...
RuntimeError: the vision encoder did not start:
```

The message after the colon is empty: the encoder exits without printing anything (compute mode was `Default`, so it is not the #1445 case).

**Cause.** `cuobjdump --list-elf` on the two binaries:

```
engine-cuda12/strata         62 sm_61, 62 sm_70
engine-cuda12/strata-vision  139 sm_70
```

`BUILD.json` then listed `archs [61, 70]` and `vision: gpu`, while `strata-vision` still had sm_70 code only.

**Workaround.** Rebuilding the encoder with the same call as `Dockerfile.v100` (`setup.cmake_build(ROOT/"tools"/"vision", ..., "strata-vision", [..., "-DSTRATA_VISION_CUDA=ON", "-DCMAKE_CUDA_ARCHITECTURES=61;70", ...])`) and copying it to `engine-cuda12/` fixes it; the server starts with vision on both P40s.

**Suggestions.** When setup recompiles the engine for a new architecture, rebuild `strata-vision` for the same list (or check its architectures, e.g. via `vision_src`/archs in `BUILD.json`); and when the encoder exits without a line, name the likely cause (no kernel image for this GPU) instead of an empty message.

Environment: Strata 0.1.44.1, CUDA 12.8 toolkit, driver 580.178.04, Linux 7.0, Docker. Benchmark of this setup: #2854.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.