#5796 [Engine] Tenstorrent Blackhole p150a card(s) as an engine

open · @simbilod · 3 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux

说明

**Question**: should this be upstreamed here, or live in a separate repository?

**What I did**: I was able to run Qwen3.8-Flash-Next on a single Tenstorrent Blackhole p150a card (32 GB) together with the CPU; currently working on other models.

The card runs the dense layers, attention and the MTP draft layer, and the CPU runs the routed experts. It speaks the line protocol of strata --serve, so Strata's own serve/server.py serves it unchanged: the chat template, tool calls, reasoning, streaming, /health and /metrics. The CPU expert rows and the suffix drafter are built from Strata's own sources. The kernel work could presumably serve as a base for other large MoE models.

**Benchmark**: a Linux, UD-IQ4_XS, 128K context build, comparison on the same machine between the p150a and a more standard GPU

| Flash-Next UD-IQ4_XS  | 1 x p150a + beefy RAM/CPU | 1 x RTX 4090 + the same CPU/RAM |
| --- | ---: | ---: |
| Prefill 8K / 32K / 128K | 1,244 / 1,558 / 1,510 tok/s | 1,599 / 1,706 / 1,981 tok/s |
| Decode (with MTP speculation) | 60-78 tok/s | 124-138 tok/s |

This decode speed ratio is somewhat expected given the memory bandwidth ratios of the cards.

Logits are checked against llama.cpp on reference prompts up to 64K. At 64K, the card's KL from llama.cpp was 0.027 against Strata's 0.010 (2026-11-08), so the card is further from llama.cpp than Strata is.

**What is implemented** (about 32K lines, written for this model -- could use cleanup):
- Card kernels, about 90 files and 16K lines of C++. They are compiled when the engine starts and run through ttnn.generic_op, so tt-metal needs no fork and no rebuild. Where ttnn already has an op that does the same work, the engine uses it (for example sparse_sdpa for long-prompt attention).
  - Decode: each DeltaNet layer runs as one persistent program on all 110 cores, attention as two, plus the hyper-connection mix, the PLE block, the head and the MTP draft layer.
  - Expert weights: decoded from their GGUF codes on the card's vector unit (IQ3_S, IQ4_NL, Q8_0), with no requantizing.
  - Prefill: causal conv, grouped experts, the attention selection (indexer score and an exact top-512 radix select), and the gate and combine kernels.
- Host side, about 14K lines of Python. The model build, the prefill, speculative decoding with the MTP layer, prompt checkpoints, conversation snapshots, and the engine's line protocol.
- CPU lane, about 2K lines of C++. The routed experts on AVX-512, built from Strata's i-quant rows, plus the card-to-host link.

**Limits today**:
- Linux only, one p150a, Flash-Next UD-IQ4_XS only.
- The engine runs in tt-metal's release container, so the user needs docker.
- One-time root setup: tt-kmd, hugepages, and the card's IOMMU group in DMA-FQ mode (a udev rule).
- Some settings are still tuned for my machine: the CPU lane's threads, and the split between card and CPU.

**What I propose:**
- Opt-in only. It would not touch the CUDA, HIP, SYCL or Metal code, and the default path would stay byte-identical.
- A launcher like sycl/serve/strata-sycl.sh that runs the engine in the container.
- A docs/TENSTORRENT.md with the setup steps and the numbers above.
- Tests through the engine's --fake mode, which runs without a card.
- Split into several focused PRs: launcher and engine, then setup checks in strata launc

I'd like the first version to take a list of cards, so supporting multiple cards later does not mean rewriting the engine.

**Questions**:
1. Do you want this in the Strata repo, or would you rather it lived outside and Strata
2. Only I can run it on hardware, and CI can't test it. Is that acceptable if each PR comes with logs and exact commands from my card?
3. Should I start with one card, or wait until multi-card support works?
4. Would one quant (UD-IQ4_XS) be enough for a first version, with Q2_0 and the other IS

**What do you do today instead?**

I run the card engine from a separate project, behind Strata's own server.

本站相关内容

相关页面的快捷入口。