#5592 [Engine] Jetson AGX Thor (sm_110): the aarch64 CUDA engine carries machine code for it, and the Terminal runs it (#4559)

open · @DobricLilujun · 0 comentários · No GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux

Descrição

Supersedes #5582 (which was closed when the branch was renamed): this version is current with `main` (merged, not
rebased), every commit is authored by the GitHub account opening it, and the eight points from the Copilot review on
#5582 are addressed: the README pack row is limited to the ARM64-feed packs; the settings ladder is rebuilt run by
run with the honest same-session switch A/B (the cross-session decode +3-8% claim is withdrawn); the A/B files have
an arm key marking what is proven and what is reconstructed, and `tools/ab_engine.py` now writes a manifest and
per-record arm specs so new runs are self-describing; every referenced raw artifact (outputs, provenance, the paired
clock suites, the dgx base files, the MAXN depth log) is inside the report; the docs' not-run list matches reality;
and the DeepSeek planner line names the missing per-set speed estimate instead of saying the Thor never ran.

## The problem, and the result

The Jetson AGX Thor is the second ARM64 Linux computer on Strata's feed: its GPU is `sm_110` (compute capability
11.0), not the DGX Spark's `sm_121`. CUDA keeps each card its own machine image: the `linux-aarch64` engine
package (`sm_121, compute_121`) cannot run on a Thor, and `engine_component.arch_covers` correctly refuses it
(different major). So on a Thor the Terminal installed but `strata launch` found no engine (issue #4559).

After this PR the aarch64 release package carries machine code for both cards - an `sm_121` cubin, an `sm_110`
cubin and `compute_110` PTX - built by the same one-line recipe on either machine, and the whole install path
(install.sh note, feed selection, setup, the driver floor) accepts a Thor. All of it was built and run on one
real Jetson AGX Thor.

## The approach

Compile level:
- `spark/build_engine.sh`: without `ARCHS` it now asks the machine (nvidia-smi's compute capability): a Spark gets
  121, a Thor 110, no answer keeps 121 (the old default). `ARCHS="121;110"` is the release shape: nvcc emits
  `sm_121` + `sm_110` cubins and `compute_110` PTX (verified with `cuobjdump --list-elf/--list-ptx`).
- `packaging/build_cuda_linux_aarch64.sh`: builds that fat shape by default and records `sm_121, sm_110,
  compute_110` in both engine packages; the NVIDIA driver floor drops to **580.00** - L4T R38 reports 580.00, and
  the old 580.65 would have blocked every Thor despite being the same r580 branch. (Flag for review: this also
  loosens the Spark's floor; a per-card floor would need a feed change this PR avoids.)
- `tools/check_cuda_spills.py`: sm_110 measured on the Thor (test build and the fat release build, max merged),
  added as `_SM110_ALLOWED` in exactly the way sm_121 was added for the Spark (`_INT3_ALLOWED`). The Thor's
  numbers sit at or under the Spark's except `expdec_down_kernel` (152 B against 136 B).
- `CMakeLists.txt`: `quantize_act_parity`'s host side compiles `-ffp-contract=off` on non-MSVC. The test's host
  references are transcriptions of the device's exact float steps; gcc on aarch64 contracts `iscale * xb[j]` into
  the magic-constant add (an x86 baseline build cannot - no FMA), and the transcription then differs from the
  device's barriered `__fmul_rn` on exact .5 ties. This is a test artifact of ARM64 hosts (the Thor AND the
  Spark), not a device issue: the same binary fails three Q8_K distributions with contraction and zero without.
- `src/core/nv_defaults.cpp` + test: an sm_110 speed table of five switches, each measured on this Thor: SPEC_COST (every case faster or equal at --spec 4; fixed windows lost at depths 2 and 6/8), HC_Q8 (drop-one: -3.1 tok/s decode, -60 prefill), PF_FUSED (drop-one: -3.8 decode, -86 prefill on a 6K prompt), HC_UPMIX (+1.8/2.7/1.3 chats, bit-identical), SIDE_MIXER=2 (+128 tok/s prefill at 6K, decode inside noise). `STRATA_SM110_PERF=0` off, `STRATA_DEFAULTS=026` excludes it, as for sm_121. End to end: the standard suite on IQ3_S with the table on gained prompt +20.3% (4K) / +9.5% (16K), time-to-first-answer -27% (4K), decode +7.6% (4K) and -6.2% (16K); sanity 12/12 both ways.

Support level:
- `packaging/install.sh`: `*GB10*|*Thor*` and the note now names both machines.
- `setup.py`: `is_thor()` (UMA + arch 110 + ARM64 Linux); the unified-memory `--mmap-experts` placement applies to
  the Thor (same one-memory reason as the Spark); the Spark's per-part speed settings stay Spark-only.
- `serve/server.py`: `sm_110` joins `sm_121` as a chip with no separate VRAM figure.
- `strata_control/dsv41_fit.py`: the one-pool wording applies to a Thor too, and it prints "Not run on a Jetson
  AGX Thor yet" instead of claiming the Spark's measured speeds.
- comments/docs in `strata_control/targets.py`, `doctor.py`, `strata_cli/update_cmd.py`.

Tests: `tests/terminal/test_linux_aarch64.py` (fat archs cover both; sm_121-only never covers a Thor; install.sh;
a Thor DeepSeek plan says one-pool + not-measured), `tests/terminal/test_release_matrix.py` (a Thor PC picks the
aarch64 cuda13 package; an x86 Thor-class card does not; sm_86-on-ARM and the old driver still refuse).

Docs: new `docs/JETSON_THOR.md` (all its numbers come from this machine), the hardware row in `docs/INSTALL.md`,
the cross-link in `docs/DGX_SPARK_LINUX.md` (the Spark keeps its measured-only claim), and the fat cuda13 row in
`packaging/README.md` / `packaging/ENGINE-COMPONENT.md`.

## Validation (all on one Jetson AGX Thor: L4T R38.2.2, driver 580.00, CUDA 13.0.48 sbsa, Ubuntu 24.04.4, aarch64; main @ ffa80d73)

Passed:
1. `ARCHS="121;110" spark/build_engine.sh` end to end: 179/179 targets; `strata 0.1.56`, `strata-dsv41 0.1.56`;
   `cuobjdump`: 70 sm_121 + 70 sm_110 cubins, 70 compute_110 PTX modules in `strata`.
2. ctest with the engine built `-DCMAKE_CUDA_ARCHITECTURES=110`, GPU in place: **193 of 195 pass** on the Thor
   (the CUDA kernel parity suites among them; `cuda_spill_guard` passes against the new `_SM110_ALLOWED`).
3. `tests/terminal`: 33/33 in the two aarch64 test files including the new Thor cases; the whole folder: 2426
   passed, 65 failures that reproduce IDENTICALLY from unmodified `main` on this machine (environment), 66
   skipped. `tools/test_setup_*.py`: 56 of 61, the same 5 failures on unmodified `main`.
4. Container path (arm64 CUDA 13.0 containers are NVIDIA's own recommendation for Thor): inside
   `eugr/spark-vllm` (CUDA 13.0, Ubuntu 24.04, `--runtime nvidia`) the fat engine prints its versions, `cuobjdump`
   there counts the 70/70/70 code sets, and a `sm_121 + sm_110` CUDA program compiled inside the container runs
   on the Thor (20 SMs, cc 11.0). (NVIDIA's `nvidia/cuda:13.0.0-devel-ubuntu24.04` pull stalled on this box's
   docker daemon mid-layer; not retried on another host.)
5. `setup.py --check` on the Thor: `GPU: NVIDIA Thor, unified memory (117 GB usable by the GPU), compute capability 11.0, driver 580.00`
   (and on the branch, the ARM64 note now names the Thor). `strata doctor` runs; its `--live` probe needs a saved profile (no model on
   this machine, see Not run).
5b. `COMMIT=<hex> packaging/build_cuda_linux_aarch64.sh` end to end ON the Thor: both engine packages built, `package_engine.py
   --verify` and `binary_deps.py` pass on each, and `engine_component.resolve` on this machine (sm_110, driver 580.00) selects that
   package (archs `sm_121, sm_110, compute_110`, floor 580.00); the packaged `strata` runs.
6. Reproduced in Docker (compatibility + reproducibility): the same chain built from scratch (source = `git archive` of this branch,
   container adds only apt `cmake ninja-build git python3`) inside the official `nvcr.io/nvidia/cuda:13.0.0-devel-ubuntu24.04`,
   `--runtime nvidia`. Its `nvcc` is the same `V13.0.48 / compiler.36260728` as the Thor's. The container's fat binary matches the
   host build function-for-function: same size, same kernel name set, same multiset of per-kernel REG/STACK/SHARED across all 140
   cubins + 70 PTX modules (`cuobjdump -res-usage`, path-hashes normalized) - not byte-equal by construction (`nvcc` hashes the
   compile path into anonymous-namespace symbols; `strata-dsv41` stamps its build time). `build_cuda_linux_aarch64.sh` end to end
   in the container: both packages verify inside and re-verify from the host, and `engine_component.resolve` on the Thor picks the
   container-built package, which runs. `ctest` in the container: 190/193 (see the third environment line below).
7. Served a real model end to end IN the container (smoke, not a benchmark): inside the same official devel image
   (`--runtime nvidia --ulimit memlock=-1`) with the container-built package as the component, `strata launch --model iq3_s
   --context 32k --vision off --drafting mtp --tune skip` downloaded nothing (the 84 GB Qwen3.8-Flash-Next IQ3_S pack was already on
   disk), loaded it in ~5 min and served on :8080: `17*23` answered 391, a 450-token completion took 13.3 s (~34 tok/s), `nvidia-smi`
   showed the Thor at 98 % during generation, `strata status` reported the model ready at context 32768, four parallel slots, 105.7
   of 131.9 GB used - and `ps` showed the running binary is the container-built `strata --serve` from the package folder.
8. The direct-install path on a Thor (what the issue asks for - a Thor user needs nothing extra): `packaging/make_local_feed.py`
   built a test-signed local feed from the two packages, and `strata update --check --feed <that feed>` on this machine says
   "0.1.56 is the one for this PC ... selected for this PC: 0.1.56 linux-aarch64-cuda-cuda13"; `install.sh`'s ARM whitelist
   matches this machine's own `nvidia-smi` name ("NVIDIA Thor" hits `*Thor*`, the warning stays silent); `setup.py --check`
   detects the card and the ARM note names the Thor. Nothing here is a Thor-only code path: it is the normal feed/install/setup
   route, now with the architecture and the driver floor that make it answer.
9. A standard benchmark on the Thor, model served from this branch's engine package: `strata bench run --suite standard-v1`
   (frozen corpus `e75eff8b...`, draft protocol) with Qwen3.8-Flash-Next IQ3_S, context 32768, MTP, four slots:
   **12 of 12 sanity checks pass** (the exact expected texts, JSON values and all four tool calls); decode 38.3 / 34.0 / 35.6
   tok/s (short / 4K / 16K prompt medians), prefill 769 / 1000 tok/s (4K / 16K), TTFT 0.96 / 5.4 / 16.4 s, no failed request;
   the optional four-at-once round aggregates 51.0 / 52.9 / 54.4 tok/s. The runner wrote its own README + results.json with
   every request; they can be attached to #4560. Qwen3.6-35B-A3B UD-Q4_K_XL finished its 22.9 GB pack download and ran the same
   suite at 32768 / four slots: **12 of 12 sanity checks pass**; decode 50.7 / 37.4 / 20.2 tok/s, prefill 1375 / 1221 tok/s
   (4K / 16K), TTFT 0.14 / 3.0 / 13.4 s, four at once ~100.5 tok/s in total. Its 128K-context + 4-slot shape is refused by the
   engine itself with exact numbers on this pool (`not enough VRAM for 4 slots at a context of 131072 (5.56 GiB each, 0.78 GiB
   free)`) - a fit line, not a crash. Report with both runs and the per-request data: `bench/results/2026-11-05-community-jetson-thor-sm110/`
   (also added to docs/COMMUNITY_BENCHMARKS.md). One machine quirk recorded there: Tegra counts reclaimable page cache against
   the GPU pool in `cudaMemGetInfo` (same root as the `expert_cache_segmented_test` miss above); freeing it as file owner via
   `posix_fadvise(DONTNEED)` - no root - starts the model cleanly.

Tegra pool accounting, measured and named (same root as `expert_cache_segmented_test` here): after an engine has run,
`cudaMemGetInfo` reports ~2-5 GiB free of 122.8 while `MemAvailable` is ~76 GB (reboot restores it; 103 orphan
`launch-demo` runtime hosts on this machine each held a small CUDA reservation). Consequences measured on the device:
a second `strata launch` dies at the engine's own slot precheck (`not enough VRAM ... 0.78 GiB free`), and the q36
engine's free-based chunk/expert sizing drops single-stream decode to 11-17 tok/s with the weights fully on 

No site

Links install, modelos, releases.