Issues / #601

#601 Linux build fails on Ubuntu 26.04 (glibc 2.43) with CUDA 12.9 (nvcc host-math conflict)

closed · @zimuhuan-code · 1 commentaires · Sur GitHub

Setup & installNVIDIA / CUDADocumentationWindowsLinux

Description

**Environment**
- OS: Ubuntu 26.04, glibc **2.43** (`ldd --version` → 2.43-2ubuntu2.3)
- GPU: 2× RTX 3060 12 GB, driver 595.84
- CUDA toolkits installed: **12.8 and 12.9** (`/usr/local/cuda-12.8`, `/usr/local/cuda-12.9`; `/usr/local/cuda` → 12.9)
- gcc/g++ available: 13, 14, 15 (default `g++` = **15**)
- Strata: **v0.1.37**, commit **db4f91a**, Linux source tree. (v0.1.38 came out the same day; its `find_nvcc()` is byte-identical to 0.1.37's, so this should still reproduce there.)

**Problem 1 — default gcc is too new for the CUDA headers**

```
/usr/local/cuda-12.9/bin/../targets/x86_64-linux/include/crt/host_config.h:143:2:
error: #error -- unsupported GNU version! gcc versions later than 14 are not supported!
```

Workaround: `CXX=/usr/bin/g++-14 CUDAHOSTCXX=/usr/bin/g++-14 ./setup.sh …`
(a successful run prints `[ok] build tools present (CUDA 12.8)` and later `[ok] engine compiled: …`)

**Problem 2 — CUDA 12.9 host-math declarations conflict with glibc 2.43** (appears once #1 is fixed)

```
/usr/include/x86_64-linux-gnu/bits/mathcalls.h(83): error: exception specification is
incompatible with that of previous function "cospi" (declared at line 2601 of
.../crt/math_functions.h)
```

… same for `sinpi`, `rsqrt`, `cospif`, `sinpif`, `rsqrtf` ⇒ `6 errors detected in the compilation of "t.cu"`.
**CMake's compiler probe is itself the first CUDA compile, so it fails there** —
`Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed` — i.e. nothing of Strata is compiled.

Minimal reproduction — no Strata code needed:

```bash
printf '#include <cuda_runtime.h>\nint main(){return 0;}\n' > t.cu
/usr/local/cuda-12.9/bin/nvcc -ccbin=/usr/bin/g++-14 -c t.cu -o t.o   # 6 errors
/usr/local/cuda-12.8/bin/nvcc -ccbin=/usr/bin/g++-14 -c t.cu -o t.o   # OK
```

Tried and it did **not** help: `-D__GLIBC_USE_IEC_60559_FUNCS_EXT_C23=0`
(glibc re-defines that macro in **`bits/libc-header-start.h` L96–102**, so a command-line define is overwritten).

**Why the obvious workaround fails**

`find_nvcc()` (v0.1.37, `setup.py` L701–722) builds its candidate list from `shutil.which("nvcc")`
**plus** every toolkit under `/usr/local/cuda*` and `/opt/cuda*`, then keeps the **highest version**
(the loop comment says *"every toolkit found; the newest wins"*). So pointing `PATH` at 12.8 does not
help — 12.8 is found and evaluated, but loses the version comparison to 12.9.

**Workaround we used** (patch shown **as an illustration**, applied to v0.1.37's `find_nvcc()`):

```python
if os.environ.get("STRATA_NVCC"):          # our local patch
    cands = [os.environ["STRATA_NVCC"]]    # replace the whole candidate list
else:
    cands = [shutil.which("nvcc")]
    cands += [str(p / "bin" / "nvcc") for p in sorted(Path("/usr/local").glob("cuda*"), reverse=True)]
    cands += [str(p / "bin" / "nvcc") for p in sorted(Path("/opt").glob("cuda*"), reverse=True)]   # Arch (#46)
```

then `STRATA_NVCC=/usr/local/cuda-12.8/bin/nvcc CUDAHOSTCXX=/usr/bin/g++-14 CXX=/usr/bin/g++-14 ./setup.sh …`

**Suggestion (any of these would unblock new Linux distros)**
1. Document "on glibc **2.43** use CUDA ≤ 12.8" in TROUBLESHOOTING / AI_SETUP (we only tested 2.43; 2.41/2.42 untested);
2. Add `--nvcc <path>` (or honour `STRATA_NVCC`) — the codebase **already filters toolkits by version** for the
   Pascal/Volta path (`find_nvcc(below=(13, 0))`), so a user-facing override fits the existing design.
   (`CUDA_HOME` is not read — only Windows-style `CUDA_PATH` — so on Linux there is currently no way in.)
3. ⚠️ **The pinned llama.cpp already documents this exact failure and gives the fix**:
   `third_party/llama.cpp/docs/build.md` §*"Fixing Compatibility Issues with Old CUDA and New glibc"* (L231–259)
   shows the same `cospi` / `math_functions.h` error and patches the CUDA headers to add `noexcept (true)`
   to `cospi`/`cospif`/`sinpi`/… Strata pins llama.cpp `3cf03257f219…` (`setup.py` L92), so the conflict and its
   upstream-sanctioned workaround are already in the dependency tree — worth linking from TROUBLESHOOTING.

Sur le site

Liens install, modèles, releases.