Sujets / Server & API
Server & API
Auto topic: Server & API
Issues
- #6541 [Engine] strata-hy4 0.1.64: EXCEPTION_ILLEGAL_INSTRUCTION (AVX-512 EVEX) on Arrow Lake-HX — current workaround disables the 1.5x AVX-VNNI cold-expert kernels
- #6259 [Other] serve: web UI static assets randomly fail (connection reset / empty reply) with parallel connections; `Bad file descriptor` in serve.log
- #6253 [Engine] macOS / Metal: open contributions, rebased on v0.2.16
- #6542 [Engine] Findings on Apple M5 Max 128 GB: Gyro-L and Hy4-Flash for coding agents on v0.2.18 (+ PRs #6538, #6539, #6540)
- #5939 [Engine] Windows CUDA12 0.1.59: Tesla V100 GPU is lost during single- and dual-GPU inference
- #6530 [Engine] Hy4-Flash SQ-64 (strata-hy4 0.1.64): long reasoning falls into an endless repetition loop (2 of 2 attempts)
- #6533 [Engine] A branching follow-up re-reads the whole prompt when the final turn is longer than 1024 tokens
- #5893 [Engine] API key changes when unloading and reloading a model, invalidating existing API clients
- #6521 [Benchmark] R9700 + Threadripper PRO 3975WX: Hy4-Flash at ~30 tok/s (SQ-64, 4K) vs. 62 tok/s in the 0.2.18 notes
- #6522 [Engine] Ollama-converted GGUF fails at load: qwen35moe.attention.head_count_kv is a per-layer array
- #6526 [Engine] gfx1200 (2x RX 9060 XT, ROCm 10.1): peer-GPU fused WMMA prefill, RDNA4 MMVQ load_int_b2/Q6KTraits, and hipBLASLt fallback notes
- #6438 [Engine] Strata 2.17 + Lazarus 4.8 + Lazarus MCP Assistent not working ?
- #6339 [Engine] 0.1.62: --lookup-max-t past 8 with --draft dflash2 ends the request (dflash2: draft rows out of range); a small fix, +11..13 % on file rewrites
- #5614 [Engine] RX 9070 XT gfx1201: 0.1.58 verify unspecified launch failure triggers MES REMOVE_QUEUE GPU reset
- #6518 [GUI] Model picker shows "Download size" for a fully downloaded Qwen-family model (IQ3_S); Gyro-S in the same folder shows "Already on this computer"
- #6511 [Benchmark] 2x Tesla P40 (sm_61) split: 0.2.17 gives +9% decode, but 220K prefill is still ~5% under the 0.2.13.1 MMQ path (same-day A/B)
- #6431 [Engine] 0.1.63 STRATA_L2_PF=1 default regresses NVFP4 batch decode by 8-12% on 2x RTX 5090 with PCIe-streamed experts
- #6513 [Engine] Linux HIP: engine crashes at start (SIGSEGV in libamdhip64) with CachyOS's hip-runtime-amd 10.1 v3/znver4 builds; Arch's build works
- #6232 [Engine] 384K compaction re-reads 264K tokens (456 s): content-addressed, position-independent segment cache for near-zero-reread compaction
- #6509 [Engine] Engine 0.1.55+ decodes 35–55% slower than 0.1.52 on RTX 3080 Ti + RTX 4070 (Windows): decode expert-cache hit falls from ~92% to 55–70% although the cache gets ~40% more slots
- #6507 [Engine] q36: one early byte change costs the whole prompt — all-or-nothing match_len has no GDN state to step back to (0 reused + 38026 read, 108 s)
- #6074 [Engine] 0.1.60 prompt kernels: -13% at 20,000 tokens on a 16 GB card, +8% at 40,000, flat at 16,000
- #5839 [Engine] 0.2.13 quoted EOS still ends agent turns with reasoning-only stop, with opt-in recovery results
- #6497 [TUI] Strata uses more threads that settings says
- #6327 [Engine] Compaction re-reads 331K-342K tokens five times (19.5-20.1 min each): the cache budget is the blocker (PR #4338)
- #6495 [Engine] Layer-major streaming prefill on Metal
- #6493 [Engine] serve loop: GENI prompt can write past the position table; --cvec offsets unchecked (static analysis)
- #6491 [Other] Ray, an assistant and coding agent built on Strata: would any part of it help the app?
- #5787 [Engine] serve: python.exe dies natively under long agent sessions — every dump faults inside CPython on the same nested `any(genexpr)` path (`serve/frontend.py` `_has_tag`)
- #6487 [Engine] 0.1.62: --vision cpu fails deterministically at startup with "kv_stream: reset: an illegal memory access" (sm_86, Swift IQ3_S) — --vision gpu on the same box is fine
- #6478 [Engine] Expert-cache hit rate and decode throughput regress from engine 0.1.52 to 0.1.62/0.1.63 on two 11 GB Turing cards (Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS)
- #6404 [Engine] gfx1030 (RX 6800 XT, Windows): 0.1.63 reads prompts 2x faster than 0.1.60 but decodes 11 % slower — same config, A/B on one host
- #5672 [Engine] gfx1100 (RX 7900 XTX) --adapt-async 1 results in timeout followed by a display-driver reset during long generations
- #5943 [Engine] --prefill auto keeps a 5250-slot lend region on a fully-resident 3x 32 GB split (0.1.58 kept 0): decode -26%, prefill -46%; STRATA_PREFILL_LEND_PCT=0 restores the full tier
- #5953 [Engine] Qwen3.6-35B-A3B fails at startup on gfx1150 (Linux, ROCm 7.15): prefill gemm cublasGemmEx f16 - cuBLAS status 6
- #6467 [Engine] 0.2.17 on Windows: the CPU picture-reader fallback dies with "Bad file descriptor" (the encoder log fd is read as a path)
- #6463 [Engine] Claude Code's small side requests evict the agent's parked conversation (default conversation cache)
- #6461 [Engine] A tool parameter named `function` is dropped even when the tool declares it (Playwright MCP `browser_evaluate`), patch included
- #6460 [Engine] Claude Code auto-compaction returns an empty summary near the context limit unless `anthropic_thinking` is `on_request`
- #6459 [Engine] Codex CLI auto-compaction fails on /v1/responses: `compaction_trigger` input items are rejected (400)
- #6448 [Engine] Scope: plan an integrated Intel GPU (Arc B390, Panther Lake) as one pool of memory with the system in the SYCL engine
- #6456 [Engine] String tool-call arguments are emitted with \uXXXX escapes (ensure_ascii=True in OutputParser._scan)
- #5678 [Engine] Qwen3.8-Flash-Next IQ3_S on sm_121 (unified memory): prefill stalls at 33,372 of 33,379 tokens, just past the 32,768 chunk; STOP not honoured, restart never finishes
- #6121 [Engine] Non-finite logits (#879) with Unsloth UD-Q4_K_XL native on RTX 4090 — deterministic at 267 tokens; IQ4_XS and Q5_K_XL clean on same engine
- #6208 [Engine] 0.2.15 decodes 28-36% slower than 0.2.7 on RTX 3060 12 GB: --expert-cache auto starts with a much worse resident set (same pack and profile)
- #6445 [Engine] Abort on UR_RESULT_ERROR_DEVICE_LOST at the verify window (Intel Arc Pro B70, Linux)
- #6077 [Engine] PLE: a VRAM hot tier for n-gram table rows, and stop pinning what is already resident
- #5995 [Engine] Exit code 0xC0000409: stack buffer overrun on V100 + RTX A4500
- #6118 [Engine] Cold 50K prefill regression: 0.1.52 → 0.1.59 → 0.1.60 on RTX 3060 12GB (−16.5%)
- #6437 [Engine] Replies end mid-word with finish_reason "stop" at ~140K context (int8 KV, MTP on)
- #6436 [Engine] Park the engine: give all VRAM back between requests, keep the process, the context and the conversation
- #6426 [Engine] --layer-split auto with DFlash2 on two cards: the noisy PCIe ratio around 4.0 picks a placement with no expert room, and the drafter is dropped
- #6423 [Engine] 0.2.17 sm_120 defaults regress prefill and cache-reuse TTFT on a mixed-arch layer split with the 12.0 card first
- #6417 [Engine] 0.1.63: after one picture through the elastic image encoder, the expert cache can grow back a segment short and --prefill auto:32768 sticks at 16384 (60K cold prompt -37 %)
- #6413 [Benchmark] v0.2.17 on 2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K: same speed as v0.2.16, `--pipeline-windows 2` no longer produces `!!!!` (12 of 12)
- #6392 [Engine] Linux HIP RX 7900 XTX (32 GB RAM, IQ2_XS low-RAM): engine hangs within 4-10 min of sustained serving, verify timeouts, also on 0.1.63
- #5636 [Engine] sm_70/75: verify-window capture dies at `copy_i32_from_mapped launch: invalid argument` on first request
- #5876 [Engine] KV reservation before expert placement starves the expert cache on small cards
- #6156 [Engine] Qwen3.6-35B-A3B on 2x RTX 3060: qwen35moe ignores multi-GPU flags — patch adds a card-1 expert tier, 37.6 → 74.8 tok/s (+97%)
- #6397 [Engine] Engine cannot use more than 8 videocards
- #6052 [Engine] Equal prompt chunks skip periodic conversation checkpoints (--prompt-cache-every)
- #6403 [Engine] 12 GB of VRAM on an RTX 5080: --prefill auto picks a 7,936-token chunk where 8,192 fits; 16K prompts read in three pieces, 16-30 % slower
- #6402 [Engine] serve: /v1/models reports the context window but no output-token limit - clients guess the answer cap
- #5756 [Engine] Tool-call value truncated when the model skips a parameter's own </parameter> after a nested pair (Qwen3.8-Flash-Next)
- #6227 [Engine] Windows CUDA 0.1.60: a start stalls at the prompt-path init (no `strata hc` line), holds 23 GB of VRAM and the engine cannot be terminated
- #6203 [Engine] serve/server.py engine_args() ignores hand edits to args without user_args
- #6171 [Engine] 0.2.15 window warm-up with STRATA_ROUTE_PRIOR: the first request after a start often ends in the middle of a number
- #6136 [Engine] 0.1.60: prompt + appended tail re-reads a whole equal chunk (append 6.4x slower than 0.1.54, first read 7.5% faster)
- #6130 [Engine] RX 7900 XTX (gfx1100, Linux), 0.2.15: picture text misread ("STRATA 412" for "STRATA 42") after earlier requests on the default HIP build; the -DSTRATA_MMQ=ON build reads it right
- #6128 [Engine] 0.1.60 on Windows: with --vram-reserve-mib 700 the expert cache grows past the auto size and decode drops to 0.2-42 tok/s on an RTX 4090
- #6110 [Engine] explicit --layer-split reloads CUDA0 dense weights a second time at startup (0 MiB freed)
- #6086 [Other] Terminal: shell tab completion for the strata command
- #6082 [Engine] PLE residency policy locks the n-gram table on a 96 GB Windows PC, then the #2090 guard cuts the prompt chunk 21,248 -> 8,192: prompts 9-14% slower, conversation cache off
- #6022 [Engine] A tool-calling model can hallucinate a tool name and leak raw tool-call markup into the reply
- #6019 [Engine] Rare non-finite logits (#879) with native IQ2_XS and IQ3_XXS packs on sm_86, engine 0.1.59.1 (none with Q2_0 on the same flags)
Pull requests
- #4310 [Engine] session files: format v2 with per-stage save/restore under --layer-split
- #6520 [Engine] Inkling-2-Mini from the official NVFP4/MXFP8 checkpoint, expert offload and CPU experts for 8-16 GB cards
- #6315 [Engine] serve: a tool call written in a shape its schema does not declare is read as the call it meant (#6285)
- #6210 [Engine] Opt-in P2P q8_1 decode push, fused P2P down-projection, and GPU peer sync flag
- #6529 [Engine] Volta sm_70 decode optimizations, zero-VRAM in-place Q8_0, NVLink split head, and volta-forever detection
- #6451 [TUI] strata launch --model-path: name Swift's shard 1 as --ple-gguf, where its n-gram table is
- #6546 [Engine] Opt-in resize_for_overwrite and parallel host memcpy for kv_mode != 0 park/restore
- #6544 [Engine] Fix O(N^2) scan in getline_fd and avoid full-line copy in BGEN/parse_i64_list
- #4878 [GUI] Keep quiet desktop chat streams alive
- #6439 [Engine] Persist the SSD conversation tier across restarts when the model fingerprint matches
- #6305 [Engine] fingerprint the resolved MTP files, so the SSD conversation tier stays on for GGUF and rt drafters
- #6540 [Engine] Hy4-Flash: SQ-96 starts on 128 GB Macs, --info no longer crashes, session reuse on by default, reasoning_effort max in configs
- #6539 [Engine] Hy4-Flash (Metal): resume at the longest common prefix, several conversation slots, SSD tier kept across restarts
- #6538 [Engine] SSD conversation tier: fingerprint the resolved MTP files and keep the tier across restarts (v0.2.18)
- #6534 [Engine] sycl: keep the branch point a follow-up can reach
- #4307 [Other] serve+webui: unload that works — pre-staged checkpoints, polite waits, honest answers
- #4306 [Other] serve: session ladder — conversations survive an engine unload
- #3553 [Engine] let a HIP helper read pinned expert rows in place
- #3552 [Engine] skip file-tier experts while a helper GPU is attached
- #3551 [Engine] add an opt-in CUDA and HIP hybrid build
- #4551 [Engine] session-file v3: incremental append saves under --session-save-append
- #3773 [Engine] q36: MTP drafts for sampled requests behind STRATA_Q36_MTP_SAMPLED (stacked on #3678)
- #6536 [Engine] fix #3354: keep default API requests greedy under MTP, gate sampled drafting behind STRATA_Q36_MTP_SAMPLED
- #6535 [Benchmark] RTX 5060 Ti / Qwen3.8-Flash-Next / IQ3_S / strata-qwen38-thinking-v1-draft.1
- #5897 [Engine] serve: let the "vision" section carry its own "env" for the image encoder
- #3800 [Engine] Token mask for JSON response formats: MTP drafts survive constrained decoding
- #6506 [Engine] add Inkling 2 model family and dual-GPU looped MoE engine support
- #3607 [Engine] native Windows support for Intel Arc SYCL, rebased onto 0.2.18 (engine 0.1.64)
- #5820 [Engine] Multiple API keys with per-key concurrency limits and reserved capacity
- #6365 [Engine] Prefer FP32 TLA sparse prefill on Battlemage in builds with local TLA headers
- #6514 [Benchmark] RX 7600 XT (gfx1102): ROCm 7.2.4 vs 10.1, and the CachyOS ROCm 10.1 crash
- #4928 [Engine] long --lookup-max-t windows over a layer split (opt-in, single conversation)
- #6510 [Engine] --experts-wex4: read the routed experts from a checked WEX4 container (opt-in, file tier)
- #4619 [Engine] DS4_LOOKUP: draft a round from the request's own history (opt-in)
- #4376 [Engine] dsv41: the WMMA expert kernels compile on pre-Volta cards
- #6508 [Engine] q36: resume from the longest common prefix with GDN state checkpoints — 0 → 12,288 reused, 37.1 → 2.9 s
- #6176 [Engine] q4u32: the 4-bit KV cache with a fitted step as the default on RDNA4
- #4255 [Other] serve: refuse a port another program holds on 127.0.0.1, and say which program
- #6492 [Other] tests: the first VisionTempFiles class was shadowed and never ran
- #6490 [Engine] Load pack weights directly into shared Metal storage
- #6486 [Engine] serve: an mcp__ prefix added to a plain tool, or dropped from an MCP tool, is that tool
- #6419 [Engine] autonomous hardware-synchronized dual-GPU decode window (P2P zero-wait) [Draft speed Lab 230 tok/s+ RFC]
- #6481 [Engine] keep the baked qwen think block when preserve_thinking is unset
- #2965 setup: a separate opt-in layers 8-33 projection vector for the original Qwen3.8-Flash-Next (any quantization)
- #6477 [Benchmark] Arc Pro B60 / Qwen3.8-Flash-Next / IQ3_S / strata-qwen38-thinking-v1-draft.1
- #3957 [Engine] serve: a long conversation survives an engine restart - parked conversations saved at a graceful stop and loaded at the next start, to files or (Windows) kept in RAM with no disk writes (opt-in STRATA_PARK_DIR, STRATA_PARK_RAMLIKE=1; layer split included)
- #6474 [Engine] serve: keep a restarted engine's batch slot state out of old request cleanup
- #6471 [Engine] serve: send STOP when closing a batch admission mid-prompt
- #5476 [Engine] gfx103x decode: opt-in one-warp MMVQ for K <= 4096 and a per-format expert layout table (STRATA_MMVQ_NW1, STRATA_EXP_TABLE)
- #2843 [Engine] Slot checkpoints for clients that rewrite the end of their history (3/3, depends on #2527 and #2528): STRATA_SLOT_CHECKPOINTS, STRATA_BRANCH_CHECKPOINT, BCHECKS
- #2528 [Engine] Slot placement policy for STRATA_SLOT_CACHE_AFFINITY (2/3, depends on #2527): protection by history length, retention scores, value decay, no wait
- #2527 [Engine] Keep conversations in pipelined batch slots (1/3): active-row windows, park=, cache-aware placement behind a policy interface
- #1614 [Engine] shared KV pool for all lanes (--kv-pool-tokens): at 2 x 256k the first token comes 2-5x sooner and lane moves ~10x faster; off = identical to main
- #6464 [Benchmark] Strix Halo 64 GB unified memory (Ryzen AI Max+ 395)
- #6453 [Engine] SYCL: an integrated Intel GPU is planned as one pool of memory with the system (Arc B390)
- #6452 [Engine] serve: an on-demand CPU picture reader starts without asking the expert cache for VRAM
- #6391 [Other] verify the context patch on reused desktop Harness stages
- #6394 [GUI] preserve long-running desktop agent sessions
- #6458 [Engine] STRATA_FILL_STAGED: copy mapped blobs to the GPU via pinned buffers
- #6455 [Engine] SYCL: an Xe3 GPU (Panther Lake, Arc B390) runs the Battlemage kernels; STRATA_INTEL_GEN sets the class
- #6454 [Engine] serve: an integrated Intel GPU's memory is the system RAM in the Monitor and the free-memory checks
- #4334 [Engine] STRATA_HOST_TIMELINE host-timeline split + summary-time draft-profiler flush (opt-in, measurement only)
- #3919 [Engine] kv: size the resident ring from the run's depth (opt-in, STRATA_KV_RING_ADAPT)
- #3729 [Engine] SYCL: flag Intel iGPUs as integrated so the expert cache is sized from MemAvailable
- #3648 [Engine] --peer-device host route: STRATA_PF_PEER_PIECES=1 (opt-in) moves the activations and the sums in 8 pieces (+7 % prompt on 2x V100)
- #6447 [Engine] Size Metal state after batch admission
- #6446 [Engine] Release Metal streaming resources when the window closes
- #6432 [Engine] Reuse Metal resident prefill scratch buffers
- #6427 [Engine] layer split auto with DFlash2: plan DFlash2's drafter and after-cache bind on the last stage, and keep the gate-off fallback loadable
- #6430 [Engine] q36: --kv fp32|fp16|fp8|fp4 for the K/V cache — 262,144 tokens on one 12 GiB card
- #4932 [Other] serve: preserve_thinking / preserve_reasoning reach the chat template
- #3819 [Engine] Layer split: a stage's commit without the host wait, as on one GPU (`STRATA_SPLIT_COMMIT_ASYNC=1`, opt-in)
- #3945 [Engine] serve: a park moves the checkpoint buffers it already holds instead of copying them, same image (opt-in, `STRATA_CKPT_PARK_REUSE=1`)
- #3760 [Engine] A third decode window in flight on a layer split (`--pipeline-windows 3`, opt-in): a host-side scheduler, a slot pool, per-stage undo, an 11-output draft chain
- #6288 [Engine] conversation cache SSD tier: a layer split's conversations and a kept folder across restarts (--conversation-spill-persist on)