Pull requests / #1990
#1990 Add opt-in external llama.cpp token engine
open · draft · @rajpaulsingh6-stack · 0 commentaires · Sur GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Description
## Summary Adds an opt-in `--engine llama-http` seam that lets Strata retain ownership of chat templating, tokenization, tool/API loops, request accounting, and OpenAI compatibility while an already-running llama.cpp-compatible server owns platform-specific device setup. The adapter validates `/health`, sends token IDs rather than rendered text, streams raw token IDs back, forwards a bounded set of sampling fields, and records prompt/decode/speculative-draft metrics. It deliberately does not start, stop, or unload the external runtime. This is intended as an experimental provider seam, not a claim of general native macOS or Metal support. Relates to #1968. ## Verified vertical slice Hardware: Mac Studio M2 Max (32 GB) with RTX 5060 Ti 16 GB in a Razer Core X V2 over USB-C/PCIe. Runtime: Strata `llama-http` -> Macuda/llama.cpp b11433 -> TinyGPU -> RTX target, with M2 Max Metal running Qwen3.8 Flash-Next MTP Q8_0 drafts (`draft-mtp`, max 3). Target model was Swift-Qwen3.8-Flash-Next-GSQ-RCO-abliterated Q2_0 (66.5 GB). - Exact-response smoke: `MADARA STRATA MACUDA MTP MAX3 LIVE`; 5.1 decode tok/s; 9/9 offered draft tokens accepted. - Workbook benchmark: the fused node selected the correct preserve-and-validate action for an 88-row, 14-team October 2026 NYC sports schedule; GenOffice reported 0 errors and 0 warnings; model QC ran at 3.3 tok/s with 81/120 drafts accepted. - Browser-grounded Knicks narrative: 243 words from NBA.com and ESPN facts; 4.1 tok/s with 263/420 drafts accepted. - `python -m py_compile serve/server.py` and `git diff --check` passed. ## Performance boundary In a controlled 128-token prompt, direct RTX target decode was 6.40 tok/s. Metal MTP drafting reached 5.61 tok/s with 84/126 draft tokens accepted (66.7%); CPU/RAM MTP drafting reached 4.85 tok/s with 83/129 accepted (64.3%). The host draft lanes are correct but slower on this USB4 topology, so this PR exposes the seam without making speculative decoding the default. ## Limits - No external process lifecycle management. - No Strata vision embeddings through this engine. - Tested on one macOS/Macuda/TinyGPU configuration at 1024-token context. - This draft PR contains the adapter only; benchmark artifacts and device-specific configuration are intentionally not committed.
Sur le site
Liens install, modèles, releases.