#6236 [Other] RTX 4090 / Qwen3.8: 131K Context, Performance & CUDA Stability — Field Report
closed · @Diko-Jelev · 3 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentationWindows
Beschreibung
#[Other] RTX 4090 / Qwen3.8: 131K Context, Performance & CUDA Stability — Field Report > **TL;DR:** On Windows 11, I ran Qwen3.8-Flash-Next IQ3_S through Strata on a single RTX 4090 (24 GB), upgraded the Desktop/Engine stack, tuned context/KV/speculative decoding and GPU clocks, and connected it to real coding and reverse-engineering agents. I observed **up to ~152 tok/s decode and ~3,263 tok/s prefill**, but also collected **11 fatal GPU-related errors and 20 non-finite-logit events** across changing configurations. This is a **field report, not a controlled benchmark**. I'm sharing the wins, failure modes, and a few usability requests so other Windows users can reproduce—or improve on—the results. I got into computers as a kid, earned a Bachelor's degree in Computer Science and Multimedia, and later turned photography from a hobby into my profession. Today I'm an enthusiastic *vibe coder* trying to make local AI do real work. Apparently, before the AI can reverse-engineer an app, someone has to reverse-engineer the AI setup. That someone was me. ## Why this report I wanted **Strata to be the local inference engine for a real, long-running workflow**, not just a quick token-speed demo: Craft Anything investigates a Windows program with IDA Pro, while OpenCode builds a Rust/egui application based on the findings. This stresses context retention, tool calls, sustained GPU usage, and process reliability differently from a short chat. I read the project's [community benchmark guidance](https://github.com/Niko1221/Strata/blob/main/docs/COMMUNITY_BENCHMARKS.md) and existing RTX 4090 performance reports, including [#1842](https://github.com/Niko1221/Strata/issues/1842). My numbers below are **operational observations** from an actively tuned system. I am **not** submitting them as standard `strata bench run` results or claiming apples-to-apples improvements against other reports. ## Test environment | Item | Configuration / observation | | --- | --- | | OS | Windows 11 x64, build **26300** | | CPU | Intel **Core i9-14900K** | | GPU | NVIDIA **RTX 4090, 24 GB** | | RAM | **64 GB DDR5**; memory tuning included 4800 → 6000 MT/s | | Strata Desktop | **0.2.6 → 0.2.15** during testing | | Strata Engine | **0.1.52 → 0.1.60** during testing | | Model | **Qwen3.8-Flash-Next IQ3_S** (local quantized model) | | Engine context | **131,072 tokens** (`--max-context 131072`) | | KV cache | **INT8** | | Speculative decoding | Tested **DFlash2** (including `--spec 6`) and **DFlash2-off / Suffix Drafting** (`--spec 4`) configurations | | GPU memory strategy | GPU-heavy configuration; VRAM reservation seen at **2048 MiB**, later **3072 MiB** in logs | | Workload | Long-running agent tool calls, code analysis, Rust development, large prompts | > **Version caveat:** Not every measurement below was taken with every setting listed here. The environment evolved during the investigation, and some tests were not controlled A/B runs. ## What improved The practical breakthrough was getting a **large local working context** and usable generation speed on a single 24 GB GPU. The Strata engine was configured for **131K tokens**; Craft Anything was later configured for a **122,221-token** working budget. Separately, OpenCode was also configured for **122,221**, with its own context-management thresholds. That distinction matters: **engine maximum context ≠ agent's usable context ≠ compaction/pruning threshold**. We initially saw agent sessions behave as though they were constrained around **77K–79K tokens**. Increasing the application-side budget helped the workflow, but the model context was never the only limiting factor. For example, OpenCode's DCP pruning and compaction had separate effective budgets, so it could compact earlier than the advertised 131K maximum. **Observed peaks (not standardized benchmark medians):** | Metric | Observed value | Context | | --- | ---: | --- | | Decode throughput | **~152 tokens/s** | Tuned, reduced-overclock configuration; sustained stability not yet established | | Prefill throughput | **~3,263 tokens/s** | Observed during the same broader tuning effort | | GPU load | Up to **100%** | During local inference | | VRAM use | Around **22.8 GB** in one recorded run | Close to the card's practical headroom | | GPU temperature / power | About **72°C / 294 W** | Snapshot from a measured run, not a long-duration average | A previous engine-version comparison also suggested modest prompt-processing improvements on our specific workloads. I would not extrapolate those readings to every prompt length, kernel, or model. ## GPU tuning: more clock is not always more tokens I experimented with the GPU-unlocking/tuning project I referred to locally as the **“4700 unlock”**. I don't have sufficient hardware-verification evidence to claim that this changed the number of physical CUDA cores, so I am recording it as an **experimental unlock/configuration procedure**, not as a Strata feature or proven hardware capability. One of the **reduced** overclock profiles associated with the encouraging ~152 tok/s result was: - **GPU core offset: +88 MHz** - **VRAM/memory offset: +555 MHz** These are **offsets**, not absolute clock speeds; the observed operating GPU clock in that run was around **2880 MHz**. Other, more aggressive profiles were tested during the investigation. None of these settings should be treated as universally safe, and the peak speed does **not** prove long-term stability. An earlier comparison of two other profiles showed that pushing memory harder could actually lower observed throughput (roughly **124.4 vs 118.8 tok/s** in that particular comparison). More MHz did not reliably mean more tokens. ## DFlash2 / speculative decoding and stability Our experimentation included **DFlash2** and a **DFlash2-disabled Suffix Drafting** alternative. During stability troubleshooting, DFlash2 was disabled and Suffix Drafting was tested with `--spec 4`; **other logs later showed DFlash2 enabled again**. Therefore, please don't interpret this as a claim that the final installed system is permanently running with DFlash2 off. Across the investigated logs we counted: - **11 fatal GPU-related errors**. - **20 non-finite-logit events**. - Errors including **CUDA illegal memory access**, **misaligned memory address**, and faults observed in or around speculative/draft/verification execution. **Important limitation:** this does **not** establish DFlash2 as the sole cause. Some CUDA errors also appeared without DFlash2, and the GPU was overclocked during parts of the investigation. The candidate causes include an unstable OC profile, VRAM pressure, driver/runtime behavior, and engine bugs. These need an isolation matrix—not guesses. ### Reproduction data that would make this more actionable For a follow-up, I would aim to capture identical prompts and model files under four controlled runs: stock clocks × DFlash2 on/off, then reduced OC × DFlash2 on/off, while holding context, KV type, GPU reserve, sampling, and engine build constant. Each run should record exact flags, model hash, driver version, per-run JSON/logs, and failure timing. I am not presenting that matrix as already completed. ## Lessons from the real agent workflow The most useful observation for other users is that **a 131K engine window does not automatically mean a coding agent can use 131K before pruning**. Every layer adds its own budget, reservations, and compaction behavior. It would help to surface **actual maximum / configured context / reserved output / pruning threshold** clearly in the Strata UI and API diagnostics. I also ran into integration friction with local coding tools: MCP connection status can be healthy while individual tools fail; and some Windows subprocess launchers briefly exposed console windows. Those particular faults were traced to **Craft Anything/OpenCode/plugin integrations**, not Strata's inference engine, and I do **not** want to misattribute them here. We separately pursued the upstream Craft Anything fix that sends long OpenCode prompts through **stdin** to avoid Windows **OS error 206** ([merged PR #1775](https://github.com/storytold/craft-anything/pull/1775)). For Strata itself, the biggest opportunity is **operational transparency**: make it simpler to distinguish a context-setting mismatch, a model/tool-call failure, an inference crash, a draft-decoding problem, and an unstable GPU configuration. ## Questions and requests for maintainers 1. **Recommended stable RTX 4090 / IQ3_S / 131K profile?** In particular, which `--spec`, DFlash2/Suffix Drafting, KV, and VRAM reserve settings would you use as a conservative baseline on Windows? 2. **Crash diagnostics:** Are there log signatures that clearly distinguish speculative decoding faults from invalid model output or CUDA instability caused by OC/VRAM pressure? What minimum log bundle would be most useful? 3. **Context visibility:** Could the UI/API show the configured backend context and effective runtime reservations, with a clear warning when a client/agent sets a smaller window? 4. **Benchmark contribution:** I can follow the official `strata bench run` protocol and submit a reproducible PR with raw results once a stock-clock baseline is captured. Is there a preferred test sequence for comparing DFlash2 and Suffix Drafting on sm_89? ## Closing **Strata made a genuinely useful local agent workflow possible on one RTX 4090.** The combination of 131K backend context and strong observed throughput is exciting—even if my logs occasionally read like CUDA was expressing its feelings. I'm sharing both the success and the messy bits because reproducible failures are just as useful to an open-source project as screenshots of impressive speeds. Happy to add sanitized logs, exact launch flags, model hash, and properly controlled benchmarks in a follow-up. Cheers from Bulgaria—and thanks to the maintainers and contributors!
Mehr auf der Site
Links zu Install, Modellen, Releases.