Issues / #3187
#3187 [Benchmark] engine calibrate --measure loses the whole sweep when the report directory does not exist
open · @IVoyt · 2 comments · View on GitHub
BenchmarksSetup & installNVIDIA / CUDAModels & quantsWindows
Description
## Area
[Benchmark] (measurements and reports)
## Desktop app, Terminal or source?
Source (START-HERE.bat, setup.sh or a build)
## What happened
`strata engine calibrate --measure` ran the full decode sweep (8 engine starts, 177 measurements, ~19 minutes) and then threw the whole result away with:
```
strata: report output directory does not exist
```
The output directory is only checked **at the very end** of the sweep, in `_write_new()` (`strata_control/calibration.py`), which is called after the measurement has finished:
- `calibration.py:303` — `result = execute(runner_cfg, say=say)` — the sweep, ~19 minutes
- `calibration.py:329` — `_write_new(output, _report_bytes(report))` — the write
- `calibration.py:231-232` — `if not path.parent.is_dir(): raise ProfileError("report output directory does not exist", 5)`
The report path's *existence* check is done up front (`calibration.py:291`, "calibration report path already exists"), but the check that the **parent directory exists** is at the end. A missing directory therefore costs the whole measurement.
The measured numbers survive only in the engine log, not in a report; the sweep's own output was on the console.
## Smallest steps or command that reproduce it
1. Pick a run config whose model is installed: `strata-swift-iq3_s.json`.
2. Run the measurement with an output path whose **parent directory does not exist**:
```
strata engine calibrate --profile D:\Strata\strata-swift-iq3_s.json --measure --exclusive --out D:\Strata-data\does-not-exist\iq3_s.json
```
3. Wait for the whole sweep (8 engine starts, ~19 minutes here).
4. The run ends with `strata: report output directory does not exist` and no report file.
The same happens with a path whose parent exists but is not a directory.
## Reproduction status
Reproduced locally.
## Strata version
Strata 0.2.0-beta (build 0.1.45.3), git `3fa1b524` (v0.2.0-beta-7), source checkout.
## Graphics card, memory and system
RTX 5090 32 GB - 96 GB RAM - Windows 11 (10.0.26300), i7-12700KF, driver 617.42, CUDA toolkit 13.1 (nvcc 13.1.115), Python 3.13.12.
## Model, size and settings
Swift 1.5 Qwen3.8-Flash-Next IQ3_S, 524288 context, pictures off.
## Logs
The sweep's console output (it completed; only the write failed):
```
round 1/3 PCIe share: 0 127.9 0.2 133.6 0.35 122.5 0.55 120.4 0.75 119.3 0.9 108.9 1 137.6
...
PCIe share, median of 3: 0: 129.5 (127.9-132.7), 0.2: 126.2 (119.5-133.6), 0.35: 124.5 (122.5-126.0), 0.55: 118.7 (118.1-120.4), 0.75: 119.3 (115.2-120.4), 0.9: 110.2 (108.9-113.6), 1: 136.5 (133.0-137.6) -> best 1
draft floor, median of 3: 0.3: 129.0 (127.8-138.2), 0.5: 137.9 (133.2-139.2), 0.7: 149.5 (140.7-151.5) -> best 0.7
Measuring with 11 CPU workers (restarts the engine) ...
11 workers: 154.7 tok/s
Measuring with 7 CPU workers (restarts the engine) ...
7 workers: 158.6 tok/s
Measuring with 6 CPU workers (restarts the engine) ...
6 workers: 163.0 tok/s
Measuring with 3 CPU workers (restarts the engine) ...
3 workers: 162.4 tok/s
Measuring the expert tier (the engine default) (restarts the engine) ...
138.1 tok/s
Measuring the expert tier every 1, 80 swaps, decay 0.97 (restarts the engine) ...
168.0 tok/s
Measuring the expert tier every 1, 160 swaps, decay 0.97 (restarts the engine) ...
167.7 tok/s
strata: report output directory does not exist
```
## Expected behavior
The run should refuse **before** the sweep, the way it already refuses an existing report path:
```
strata: report output directory does not exist
```
or write the report where it can.
## Suggested fix
Check the output's parent directory in `_measure_locked()` next to the existing "report path already exists" check (`calibration.py:291`), so the refusal happens in a second rather than after 19 minutes of measurement:
```python
if not output.parent.is_dir():
raise ProfileError("report output directory does not exist", 5)
if output.exists() or output.is_symlink():
raise ProfileError("calibration report path already exists", 2)
```
## Additional context
- Not a crash and not a wrong answer: the measurement itself is fine, only the report is lost. The numbers are still in the engine log (`strata-<model>.log`), and the console shows them, but there is no report for `--review-result` / `--apply-result` to consume.
- The same late check applies to `setup.py --calibrate` (it writes its own record, so it does not fail, but it also has no up-front check).
- The mistake is easy to make: the `--out` path is free text, and nothing checks the directory until the very end.
Related on this site
Quick links to related pages.