#3525 [Engine] GPU kernel crash on NVIDIA GB10 (Grace Blackwell) — 'illegal memory access was encountered' during verification

Description

## Problem

GPU kernel crashes on NVIDIA GB10 (Grace Blackwell) with "illegal memory access was encountered" during the verification step. Every inference request fails immediately after model loads. The error occurs even on a completely fresh server start with zero prior requests.

## Environment

| Item | Value |
|------|-------|
| Strata version | v0.1.39 (local build), also tested v0.1.46 official main branch |
| Build commit | Our tree: f6deb60 merge apple-silicon-metal; Official: 9ae93f7 hotfix v0.1.46.1 |
| CUDA architecture | sm_121 (tested), sm_120 (original default), sm_120+sm_121 multi-arch |
| CUDA toolkit | 13.0 (primary), 13.1, 13.2, 13.4 available |
| Driver | 595.91.07 (tested); upstream DGX Spark user used 580.142 |
| GPU | NVIDIA GB10 (Grace Blackwell), compute capability 12.1, ~96 GiB VRAM |
| CPU | ARM64 (aarch64), 24 cores (Cortex-X925 + A725 equivalent) |
| OS | Ubuntu 24.04, kernel 6.x |
| Model | Qwen3.8-Flash-Next-GSQ-RCO-Q2_0 (IQ2_XS quantization) |
| Config args | `--pack /home/harness/Strata-data/packs/q2_0 --native ...gguf --ple-gguf ...gguf --mmap-experts --prefill auto --max-context 4096 --expert-cache auto --no-pool` |
| CMake flags | `-DCMAKE_CUDA_ARCHITECTURES=121 -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc` |

## Reproduce

```bash
# 1. Clone the arm-gb10-port branch (PR #409) for ARM compatibility
git clone -b arm-gb10-port https://github.com/eelgaev/Strata-AC922.git
cd Strata-AC922

# 2. Configure with GB10 architecture
mkdir -p build && cd build
cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -G Ninja \
  -DSTRATA_ENABLE_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES="121"

# 3. Build
ninja -j$(nproc) strata

# 4. Copy to engine directory (or use setup.py path)
cp strata /path/to/engine/strata

# 5. Start server (this config is from the user's actual setup)
python3 -m serve.server --engine strata \
  --config '{"exe":"...","args":["--pack",PATH,"--native",MODEL1,"--ple-gguf",MODEL2,"--mmap-experts","--prefill","auto","--max-context","4096","--expert-cache","auto","--no-pool"],"cwd":PATH}' \
  --host 0.0.0.0 --port 8025

# Wait for "ready:" line to appear. Then:

# 6. Send any inference request
curl http://127.0.0.1:8025/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.8-Flash-Next","messages":[{"role":"user","content":"你好"}],"max_tokens":20}'
```

Also tested without `--mmap-experts` flag — same error.

## Expected

Model loads successfully and returns token completions as it does on supported GPUs (RTX 20/30/40/50 series). PR #409 (arm-gb10-port branch) reports working on a real DGX Spark with similar setup, producing ~60 tok/s decode.

## Actual

Every request fails with:
```json
{"error":{"type":"invalid_request_error","message":"verify: an illegal memory access was encountered"}}
```

Log shows the engine initializes and enters verification successfully:
```
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: Xxxx MiB of VRAM free with everything loaded
strata serve: verify: an illegal memory access was encountered
[strata] the engine reported an error: verify: an illegal memory access was encountered
```

The graph capture completes without error, but the actual CUDA kernel execution during inference triggers `cudaErrorIllegalAddress` (code 77 = "an illegal memory access was encountered"), crashing the process. The engine then restarts for each subsequent request, producing identical output.

## Evidence

- Full server log excerpt attached below (truncated at key error lines)
- PyTorch CUDA test confirms GPU tensor operations work fine (problem is Strata-specific kernels)
- Error occurs identically across three builds: sm_120 only, sm_121 only, sm_120+sm_121 multi-arch

### Log snippet (fresh start, single request)
```
ARM64 GPU-only expert verification
strata verify: window up to 2 tokens, 32.2 MiB of device buffers (100% VRAM resident: zero-doorbell graph)
strata serve: 1206 MiB of VRAM free with everything loaded
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: verify: an illegal memory access was encountered
[strata] the engine reported an error: verify: an illegal memory access was encountered
[strata] done: 0 tokens in 0 s (0.0 tok/s) (error, cancel=False)
```

### Additional evidence collected
1. Tested on both our local tree (v0.1.39 with PR #409 patches) and official main branch (v0.1.46.1) — identical crash
2. Verified `immintrin.h` header availability on ARM system — build requires portable.cpp from arm-gb10-port (included)
3. nvidia-smi confirms GPU operational: 70°C, 53W power draw, 2411 MHz clock
4. Tested with two different driver versions conceptually (current: 595.91.07; reference from PR #409: 580.142)

**Reproduction status: reproduced locally** on this machine. Confirmed on multiple build configurations.

---

### Suggested fix direction

This appears to be a micro-architecture difference between Ada Lovelace (sm_120, RTX 50) and Blackwell (sm_121, GB10) that affects SM_12x kernels beyond just the compute capability version number. Potential solutions:

1. Add Blackwell-specific CUDA kernel variants for affected paths (preferred for performance)
2. Fall back to PTX JIT compilation at runtime if sm_121 image fails
3. Update CMakeLists.txt to detect compute capability >= 12.1 and target sm_121 by default on those GPUs