#6628 [Benchmark] Quick Agent benchmark silently changes workload after the 0.2.16 layout move

open · @mmischitelli · 0 commentaires · Sur GitHub

BenchmarksSetup & installAMD / HIPModels & quantsDocumentationLinux

Description

### Area

[Benchmark] (measurements and reports)

### Desktop app, Terminal or source?

Source (`terminal/tools/strata_bench.py`, quick speed check)

### What happened

The [0.2.16 layout move](https://github.com/Niko1221/Strata/commit/4058aa782) put `strata_bench.py` under `terminal/tools/`. Its `ROOT` now resolves to `terminal/`, but the project README and docs used by [`Suite.agent_files()`](https://github.com/Niko1221/Strata/blob/main/terminal/tools/strata_bench.py#L318-L332) are at the repository root. On the new layout the Agent session silently substitutes all five source texts with `generated/part0.txt` through `generated/part4.txt`. The report does not identify which corpus was used, so the same metric names can describe different workloads.

Observed on the same machine and model, 3 sessions per run, thinking off:

| Quick Agent metric | 2026-11-15, old layout | 2026-11-22, new layout |
| --- | ---: | ---: |
| Median new input tokens per session | 12,991 | 8,208 |
| Median follow-up time to first token | 1.487 s | 0.834 s |

These runs also used different engine versions. The timings illustrate the misleading comparison, not an engine speedup or a controlled A/B. The workload change itself is reproducible without a server or GPU (below).

**Expected:** The Agent workload should not silently change with installation layout. At minimum, report the corpus/fallback identity and mark results with different corpora as not comparable. A fixed, versioned input corpus would also prevent ordinary doc edits from changing this workload. This is about the quick speed check, not `strata bench run` (standard v1 does not include this multi-turn Agent session).

### Smallest steps or command that reproduce it

From the repository root on the new layout:

```sh
PYTHONDONTWRITEBYTECODE=1 python - <<'PY'
import runpy
ns = runpy.run_path("terminal/tools/strata_bench.py")
agent_files = ns["Suite"].agent_files
print([name for name, _ in agent_files(None)])
agent_files.__globals__["ROOT"] = ns["ROOT"].parent
print([name for name, _ in agent_files(None)])
PY
```

The first line prints the five `generated/part*.txt` names with the script's normal root. The second changes only the lookup root in memory, and prints `README.md`, `docs/HOW_IT_WORKS.md`, `docs/DETAILS.md`, `docs/INSTALL.md` and `docs/MODELS.md`.

### Reproduction status

Reproduced locally: observed in the quick benchmark reports and confirmed directly by running the input-selection function.

### Strata version

Old layout: checkout `d53fa679`, engine `0.1.59.1`. New layout: installed source reported `0.1.65` (no Git checkout), engine `0.1.63`.

### Graphics card, memory and system

AMD Radeon RX 9070 XT, 16 GiB VRAM, 62.3 GiB system RAM, Arch Linux (kernel `7.3.4-arch1-2` and `7.3.5-arch1-1` respectively), local HIP builds for `gfx1201`.

### Model, size and settings

Qwen3.8-Flash-Next IQ3_XXS, 256K effective context, 1 parallel slot.

Sur le site

Liens install, modèles, releases.