#3451 Bug: NVIDIA GB10 (Grace Blackwell) GPU kernel crash — 'illegal memory access was encountered' during verification

Descrição

# Bug: NVIDIA GB10 (Grace Blackwell, compute capability 12.1) GPU kernel crash — "illegal memory access was encountered"

## Environment

| Item | Value |
|------|-------|
| **GPU** | NVIDIA GB10 (Grace Blackwell) |
| **Compute Capability** | 12.1 |
| **Driver Version** | 595.91.07 |
| **CUDA Toolkit** | 13.0 / 13.1 / 13.2 / 13.4 |
| **CPU** | ARM64 (aarch64) |
| **Strata Build** | sm_120 (default RTX 50), also tested sm_121 and sm_120+sm_121 multi-arch |
| **Model** | Qwen3.8-Flash-Next-GSQ-RCO-Q2_0 |

## Steps to Reproduce

1. Clone `teoria/Strata_arm64`
2. Build with CUDA arch targeting GB10:
   ```bash
   cmake -S . -B build -G Ninja -DSTRATA_ENABLE_CUDA=ON \
     -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc \
     -DCMAKE_CUDA_ARCHITECTURES="121"
   ninja -j$(nproc) strata
   ```
3. Start server:
   ```bash
   python3 -m serve.server --engine strata --config strata.json --host 0.0.0.0 --port 8025
   ```
4. Send any chat completion request:
   ```bash
   curl http://127.0.0.1:8025/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.8-Flash-Next","messages":[{"role":"user","content":"你好"}],"max_tokens":20}'
   ```

## Expected Behavior

The model loads successfully and returns token completions as it does on supported GPUs (RTX 20/30/40/50 series).

## Actual Behavior

Every request fails with:
```json
{"error":{"type":"invalid_request_error","message":"verify: an illegal memory access was encountered"}}
```

Log output shows the engine initializes correctly:
```
GPU 0: NVIDIA GB10, compute capability 12.1
ARM64 GPU-only expert verification
strata verify: window up to 2 tokens, 32.2 MiB of device buffers
strata serve: Xxxx MiB of VRAM free with everything loaded
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: verify: an illegal memory access was encountered
```

The graph capture succeeds ("no error"), but when the actual kernel runs during the first inference request, the CUDA runtime reports `cudaErrorIllegalAddress` (code 77 = "an illegal memory access was encountered"), crashing the engine process.

## Analysis

- The default CMakeLists.txt sets `CMAKE_CUDA_ARCHITECTURES 120` (for RTX 50 Ada Lovelace)
- GB10 has compute capability **12.1**, which differs from RTX 50's **12.0**
- Tested three builds: sm_120 only → crash, sm_121 only → crash, sm_120+sm_121 multi-arch → crash
- Disabling expert cache (`--expert-cache 0`) causes SIGSEGV (-11) instead
- The error occurs in the **verification pipeline** which exercises the same kernels used by the decode path
- This suggests micro-architecture differences between Ada Lovelace (sm_120) and Blackwell (sm_121) are not just version bumps but involve instruction set or memory layout changes that break Strata's CUDA kernels

## Requested Fix

1. Add `sm_121` as a recognized target architecture in CMakeLists.txt alongside `sm_120`
2. Either:
   - Add Blackwell-specific CUDA kernel variants (preferred for performance), OR
   - Fall back to PTX JIT compilation at runtime (easier but slower)
3. Update AI_SETUP.md to document GB10 support requirements

## Attachments

- Full engine log: available on request
- `nvidia-smi` output showing driver/CUDA versions
- PyTorch CUDA test confirming GPU is operational (tensor operations work fine — problem is specific to Strata's custom kernels)