#2386 cannot use strata in opencode through 0.1.44.1 by RTX4090

Description

Re: "The engine stopped unexpectedly" - OpenCode client
Date: 2026-10-14 (update, supersedes the 2026-10-12 report)
User: RNGMARTIN

Short version: the stall reproduces on 0.1.40.3, 0.1.43 and 0.1.44.1 - every
version I tried. Same stage, same signature, every time. It is not RAM
exhaustion and it is not the client.

====================================================================
1. The line and the 30 lines of the server window before
   "the engine stopped unexpectedly (exit code ...)"
====================================================================

Latest reproduction, 2026-10-14, engine 0.1.44.1, started via START-HERE.bat.
The server window showed, in order:

    [strata] reading the prompt: 588 tokens, 10 s so far
    [strata] reading the prompt: 583 of 588 tokens, 15 s so far
    [strata] thinking: 18 of max 32000 tokens, 22.2 tok/s, 16 s
    [strata] thinking: 37 of max 32000 tokens, 19.3 tok/s, 18 s
    [strata] thinking: 52 of max 32000 tokens, 17.4 tok/s, 19 s
    [strata] thinking: 70 of max 32000 tokens, 17.1 tok/s, 20 s
    [strata] thinking: 94 of max 32000 tokens, 18.2 tok/s, 21 s
    [strata] done: 103 tokens in 21 s (19.0 tok/s) (stop, cancel=False), expert cache 95.0% hit
    [strata] reading the prompt: 8,069 tokens, 10 s so far
    [strata] reading the prompt: 8,069 tokens, 20 s so far
    [strata] reading the prompt: 8,069 tokens, 30 s so far
    [strata] reading the prompt: 8,069 tokens, 40 s so far
    [strata] reading the prompt: 8,069 tokens, 51 s so far
    [strata] reading the prompt: 8,069 tokens, 61 s so far
    [strata] reading the prompt: 8,069 tokens, 71 s so far
    [strata] the engine stopped unexpectedly (exit code 3221226505). The engine stopped
    itself because it had stopped making progress - a hang it caught. Its log line:
    strata serve: no progress for 60 s during a request (reading the prompt (batched):
    layer 1 of the prompt chunk from token 0) - stopping the engine so the server starts
    it again (issue #29)
    [strata] done: 0 tokens in 74 s (0.0 tok/s) (error, cancel=False)
    [strata] the engine had stopped (exit code 3221226505); starting it again ...

The matching engine log (strata-iq2_xs.log, 0.1.44.1, 2026-10-14):

    strata serve: no step finished for 60 s, but the file tier is still being read
    (slow storage or low RAM): waiting up to 600 s in all (STRATA_WATCHDOG_IO_S) (#1407)
    strata serve: no progress for 60 s during a request (reading the prompt (batched):
    layer 1 of the prompt chunk from token 0) - stopping the engine so the server
    starts it again (issue #29)
    strata serve: stall report (engine 0.1.44.1): stage "reading the prompt (batched):
    layer 1 of the prompt chunk from token 0" for 62 s; 0 layers served since the last
    finished step (0 = stopped, more = slow)
      VRAM free when this request began: 1055 MiB
      expert pool: epoch 2860, batch epoch 2860: 48 of 48 jobs claimed, 48 done;
        15 of 15 workers parked, 15 sleeping; mode 0
      expert pool threads: w0=sleeping ... w14=sleeping; host idle for 62235 ms
      verify window (last window, not the current stage): 4 tokens at position 686,
        host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48
      memory: 37855 MiB resident, 60866 MiB committed, 9583 MiB RAM available;
        10662519 page faults so far
      2 s later:
      expert pool: epoch 2860, batch epoch 2860: 48 of 48 jobs claimed, 48 done;
        15 of 15 workers parked, 15 sleeping; mode 0
      expert pool threads: w0=sleeping ... w14=sleeping; host idle for 65076 ms
      verify window (last window, not the current stage): 4 tokens at position 686,
        host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48
      memory: 37855 MiB resident, 60866 MiB committed, 9580 MiB RAM available;
        10662661 page faults so far
      wrote the thread stacks to F:\openresource\Strata-main\strata-stall-32556.dmp
    strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms

Notes:
- exit code 3221226505 is 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN). It is the engine's
  own self-stop path after it caught the hang, not an OOM kill; the Windows System log
  has no OOM event.
- Pattern: the first request of the session (OpenCode's 588-token title request)
  always completes. The NEXT request (the real conversation prompt, here 8,069 tokens)
  hangs at the same stage. My own back-to-back API tests (53, 53, 8,457 tokens right
  after a fresh start) passed on 0.1.44.1 - the repro needs the title request first,
  which is exactly what OpenCode sends.
- The engine log also carries a warning on every start that names the stage:
  "WARNING: --ple-io direct measures slow here (407-455 rows/s, about 2 MB/s of random
  4 KiB reads): a prompt can spend minutes reading layer 1 and trip the stall watchdog
  (#1425, #1629)." The stall sits at exactly that stage (layer 1).
- The #1407 "file tier is still being read" grace was NOT active at the stop: the
  engine decided after 60 s that nothing was being read, and stopped itself.

====================================================================
2. Strata version, OS, GPU and VRAM, total RAM, model
====================================================================

- Reproduced on engines 0.1.40.3, 0.1.43 and 0.1.44.1 (latest release, 2026-10-14).
  Code updated to main each time; the latest test used the 0.1.44.1 release engine.
- OS: Windows 11 Home, Chinese edition, 10.0.26200, 64-bit
- GPU: NVIDIA GeForce RTX 4090 D, 24564 MiB VRAM, driver 610.60
- CPU: Intel Core Ultra 9 285K (no AVX-512; the log says the expert kernels run on
  AVX-2)
- Total RAM: 63 GB
- Model: IQ2_XS (Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS, two GGUF shards, 36.5 + 26.8 GB).
  The log says it "loads about 39 GB into RAM and locks part of it for the GPU".
  An IQ3_XXS pair is also on disk but not used.

====================================================================
3. The command line / config, especially context size and --prefill
====================================================================

Started with START-HERE.bat (menu choice 1 = iq2_xs). Engine command line exactly
as logged:

    strata.exe --serve
      --pack F:\openresource\Strata-data\packs\iq2_xs
      --native F:\openresource\Strata-data\models\IQ2_XS\Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf
      --ple-gguf F:\openresource\Strata-data\models\IQ2_XS\Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf
      --expert-profile F:\openresource\Strata-main\data\expert-profile.bin
      --expert-cache auto
      --prefill auto
      --spec 4 --spec-min-p 0.5
      --mtp F:\openresource\Strata-data\mtp\rt
      --max-context 131072
      --kv int8 --kv-resident 32768
      --pool-workers 15 --vram-reserve-mib 1500

So: --prefill "auto" (chooses the batched path, "prompt chunk auto: 8192 tokens"),
context 131072. Server listens on 0.0.0.0:8080 with an API key.
From the start-up log, things that may matter on this machine:
  - "expert arena: cudaHostRegister PORTABLE ok; large pages refused for
    35456548864 B (VirtualAlloc error 1314); using 4 KB pages"
  - "--ple-io direct measures slow here (407-455 rows/s, ~2 MB/s of random 4 KiB
    reads)" with the note that mmap was "40-100x faster on some DRAM-less NVMe
    drives but 7x slower on a SATA SSD (#2040), so it is not chosen for you"
  - "Windows commit: 56.8 GiB left of a 154.4 GiB limit (RAM + page file)"

====================================================================
4. Free RAM while it happens / OOM check
====================================================================

- From the 0.1.44.1 stall report: 37855 MiB resident, 60866 MiB committed,
  9583 MiB RAM available, over 10.6 million page faults so far.
- The 0.1.43 stall (same day as this update) had 14729 MiB RAM available.
  The 0.1.40.3 stall had 5565 MiB available.
- So the stall happens with 5.5, 9.6 and 14.7 GB of RAM free - it is not RAM
  exhaustion.
- Windows has no dmesg; the System event log shows no process killed for lack of
  memory. The exit code is the engine's self-stop (0xC0000409), not -9/137.

====================================================================
5. Does a long prompt (10-20K tokens, OpenCode scale) work from the
   web page / API?
====================================================================

- 9,357 tokens through the API: OK (read in 9.1 s at 1031 tok/s) - 0.1.40.3
- 8,457 tokens through the API: OK (6.6 s total) - 0.1.44.1
- But in the OpenCode flow, the 8,069-token request that follows the 588-token
  title request hangs at layer 1 for over 60 s with zero progress - on all three
  engine versions.
- Conclusion so far: it is not "the prompt is too long" and not "the client times
  out". It is a hang in the engine's batched prefill path, triggered by the request
  sequence (title request, then the real prompt), at a stage the start-up warning
  already flags as slow on this storage (--ple-io direct, ~2 MB/s random reads).

====================================================================
6. Is the engine dying, or is the client giving up?
====================================================================

The engine is hanging and stopping itself (the server's 60 s watchdog, issue #29).
The client (OpenCode) only sees the request error out afterwards. On every
reproduction:
  - all 15 expert-pool workers sleeping, host idle for 60+ s
  - "reading the prompt (batched): layer 1 of the prompt chunk from token 0"
  - "the GPU finished in 0 ms" when the verify window's waits are released
  - a fresh engine then loads (1-4 minutes), and the next request can hit the same
    wall again (15 stall dumps collected, 2026-10-08 through 2026-10-14)