问题 / #3187

#3187 [Benchmark] engine calibrate --measure loses the whole sweep when the report directory does not exist

open · @IVoyt · 2 评论 · 去 GitHub 看

BenchmarksSetup & installNVIDIA / CUDAModels & quantsWindows

说明

## Area

[Benchmark] (measurements and reports)

## Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

## What happened

`strata engine calibrate --measure` ran the full decode sweep (8 engine starts, 177 measurements, ~19 minutes) and then threw the whole result away with:

```
strata: report output directory does not exist
```

The output directory is only checked **at the very end** of the sweep, in `_write_new()` (`strata_control/calibration.py`), which is called after the measurement has finished:

- `calibration.py:303` — `result = execute(runner_cfg, say=say)` — the sweep, ~19 minutes
- `calibration.py:329` — `_write_new(output, _report_bytes(report))` — the write
- `calibration.py:231-232` — `if not path.parent.is_dir(): raise ProfileError("report output directory does not exist", 5)`

The report path's *existence* check is done up front (`calibration.py:291`, "calibration report path already exists"), but the check that the **parent directory exists** is at the end. A missing directory therefore costs the whole measurement.

The measured numbers survive only in the engine log, not in a report; the sweep's own output was on the console.

## Smallest steps or command that reproduce it

1. Pick a run config whose model is installed: `strata-swift-iq3_s.json`.
2. Run the measurement with an output path whose **parent directory does not exist**:
   ```
   strata engine calibrate --profile D:\Strata\strata-swift-iq3_s.json --measure --exclusive --out D:\Strata-data\does-not-exist\iq3_s.json
   ```
3. Wait for the whole sweep (8 engine starts, ~19 minutes here).
4. The run ends with `strata: report output directory does not exist` and no report file.

The same happens with a path whose parent exists but is not a directory.

## Reproduction status

Reproduced locally.

## Strata version

Strata 0.2.0-beta (build 0.1.45.3), git `3fa1b524` (v0.2.0-beta-7), source checkout.

## Graphics card, memory and system

RTX 5090 32 GB - 96 GB RAM - Windows 11 (10.0.26300), i7-12700KF, driver 617.42, CUDA toolkit 13.1 (nvcc 13.1.115), Python 3.13.12.

## Model, size and settings

Swift 1.5 Qwen3.8-Flash-Next IQ3_S, 524288 context, pictures off.

## Logs

The sweep's console output (it completed; only the write failed):

```
    round 1/3 PCIe share: 0 127.9  0.2 133.6  0.35 122.5  0.55 120.4  0.75 119.3  0.9 108.9  1 137.6
    ...
    PCIe share, median of 3: 0: 129.5 (127.9-132.7), 0.2: 126.2 (119.5-133.6), 0.35: 124.5 (122.5-126.0), 0.55: 118.7 (118.1-120.4), 0.75: 119.3 (115.2-120.4), 0.9: 110.2 (108.9-113.6), 1: 136.5 (133.0-137.6) -> best 1
    draft floor, median of 3: 0.3: 129.0 (127.8-138.2), 0.5: 137.9 (133.2-139.2), 0.7: 149.5 (140.7-151.5) -> best 0.7
  Measuring with 11 CPU workers (restarts the engine) ...
    11 workers: 154.7 tok/s
  Measuring with 7 CPU workers (restarts the engine) ...
    7 workers: 158.6 tok/s
  Measuring with 6 CPU workers (restarts the engine) ...
    6 workers: 163.0 tok/s
  Measuring with 3 CPU workers (restarts the engine) ...
    3 workers: 162.4 tok/s
  Measuring the expert tier (the engine default) (restarts the engine) ...
    138.1 tok/s
  Measuring the expert tier every 1, 80 swaps, decay 0.97 (restarts the engine) ...
    168.0 tok/s
  Measuring the expert tier every 1, 160 swaps, decay 0.97 (restarts the engine) ...
    167.7 tok/s
strata: report output directory does not exist
```

## Expected behavior

The run should refuse **before** the sweep, the way it already refuses an existing report path:

```
strata: report output directory does not exist
```

or write the report where it can.

## Suggested fix

Check the output's parent directory in `_measure_locked()` next to the existing "report path already exists" check (`calibration.py:291`), so the refusal happens in a second rather than after 19 minutes of measurement:

```python
if not output.parent.is_dir():
    raise ProfileError("report output directory does not exist", 5)
if output.exists() or output.is_symlink():
    raise ProfileError("calibration report path already exists", 2)
```

## Additional context

- Not a crash and not a wrong answer: the measurement itself is fine, only the report is lost. The numbers are still in the engine log (`strata-<model>.log`), and the console shows them, but there is no report for `--review-result` / `--apply-result` to consume.
- The same late check applies to `setup.py --calibrate` (it writes its own record, so it does not fail, but it also has no up-front check).
- The mistake is easy to make: the `--out` path is free text, and nothing checks the directory until the very end.

本站相关内容

相关页面的快捷入口。