#6409 [Engine] Qwen3.8-Next-Flash on Strix Halo AI Max 395 (128GB UMA): tuning for 2 concurrent streams at >100 tok/s each @ 200k context

open · @bobwei192-star · 2 comentarios · En GitHub

BenchmarksSetup & installAMD / HIPModels & quantsSecurityLinux

Descripción

### Area

Engine

### Your question

I run Strata on a Strix Halo (Ryzen AI Max+ 395) machine with 128GB unified memory and no discrete VRAM. Use case: CI/CD daily work — Jenkins pipeline maintenance, test-case authoring, debug and log analysis. My prompts repeat a lot of template content, and I need two streams to run at the same time.

Target configuration:

- Model: Qwen3.8-Next-Flash
- Concurrency: 2 simultaneous requests
- Context: 200k tokens per request
- Goal: sustain >100 tok/s decode per stream while both requests run at the same time

Two things I am asking about:

1. Which optimization directions are worth tuning for this hardware and workload — prefix cache reuse for the repeated template prompts, KV cache layout under unified memory, ROCm/HIP kernel settings, and scheduling strategy for concurrent streams?
2. Which metrics and test points should a benchmark on this machine cover to validate the goal above (e.g. prefill vs decode split, per-stream tok/s under concurrency, TTFT, KV/prefix cache hit rate, memory headroom at 200k)?

### Strata version

0.2.14.1 (build 0.1.60.1), from `strata --version`.

### System

AMD Ryzen AI Max+ 395 (Strix Halo), 128GB unified memory, Linux.

### What you tried

Strata 0.2.14.1 is installed here, but I have not downloaded the model or run any measurement yet. This issue is the scoping question before I set up a benchmark, so I am not reporting any numbers or claiming any speed.

En el sitio

Enlaces a install, modelos, releases.