Pull requests / #1990

#1990 Add opt-in external llama.cpp token engine

open · draft · @rajpaulsingh6-stack · 0 comentarios · En GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants

Descripción

## Summary

Adds an opt-in `--engine llama-http` seam that lets Strata retain ownership of chat templating, tokenization, tool/API loops, request accounting, and OpenAI compatibility while an already-running llama.cpp-compatible server owns platform-specific device setup. The adapter validates `/health`, sends token IDs rather than rendered text, streams raw token IDs back, forwards a bounded set of sampling fields, and records prompt/decode/speculative-draft metrics. It deliberately does not start, stop, or unload the external runtime.

This is intended as an experimental provider seam, not a claim of general native macOS or Metal support. Relates to #1968.

## Verified vertical slice

Hardware: Mac Studio M2 Max (32 GB) with RTX 5060 Ti 16 GB in a Razer Core X V2 over USB-C/PCIe.

Runtime: Strata `llama-http` -> Macuda/llama.cpp b11433 -> TinyGPU -> RTX target, with M2 Max Metal running Qwen3.8 Flash-Next MTP Q8_0 drafts (`draft-mtp`, max 3). Target model was Swift-Qwen3.8-Flash-Next-GSQ-RCO-abliterated Q2_0 (66.5 GB).

- Exact-response smoke: `MADARA STRATA MACUDA MTP MAX3 LIVE`; 5.1 decode tok/s; 9/9 offered draft tokens accepted.
- Workbook benchmark: the fused node selected the correct preserve-and-validate action for an 88-row, 14-team October 2026 NYC sports schedule; GenOffice reported 0 errors and 0 warnings; model QC ran at 3.3 tok/s with 81/120 drafts accepted.
- Browser-grounded Knicks narrative: 243 words from NBA.com and ESPN facts; 4.1 tok/s with 263/420 drafts accepted.
- `python -m py_compile serve/server.py` and `git diff --check` passed.

## Performance boundary

In a controlled 128-token prompt, direct RTX target decode was 6.40 tok/s. Metal MTP drafting reached 5.61 tok/s with 84/126 draft tokens accepted (66.7%); CPU/RAM MTP drafting reached 4.85 tok/s with 83/129 accepted (64.3%). The host draft lanes are correct but slower on this USB4 topology, so this PR exposes the seam without making speculative decoding the default.

## Limits

- No external process lifecycle management.
- No Strata vision embeddings through this engine.
- Tested on one macOS/Macuda/TinyGPU configuration at 1024-token context.
- This draft PR contains the adapter only; benchmark artifacts and device-specific configuration are intentionally not committed.

En el sitio

Enlaces a install, modelos, releases.