改动 / #2691

#2691 tools tests: the golden compare takes either platform's engine names, on both sides

open · @clothespin · 0 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIWindowsLinux

说明

## Title
Issue: none. No open issue or pull request covers this (searched `test_setup_golden`, `strata-vision.exe` and `golden`).
It is a test-only bug, separate from #2041, which is where I hit it.

## Summary
`tools/test_setup_golden.py` cannot match the committed baseline on Linux. The baseline was recorded on Windows, so its
five unsloth UD-Q4_K_XL configs hold `"<T>/engine/strata-vision.exe"`, while on Linux setup writes `strata-vision`
(`setup.py:498`, `VEXE = "strata-vision.exe" if WIN else "strata-vision"`). `normalize()` replaced only `setup.EXE`, and
only in the config it had just written - the baseline read back in `setUpClass` was compared as it was. Every
UD-Q4_K_XL case therefore differs on Linux: 8 failures, on a clean checkout of `origin/main` (`296b59d2`) as much as on a
branch. One commit, `tools/test_setup_golden.py` only; `tools/test_setup_golden.json` is untouched.

Re-recording with `--write` is not the fix. On Linux it changes exactly those five lines and nothing else, which makes
the baseline Linux-spelled and moves all five onto Windows.

## What changed
`canonical()` maps either platform's spelling of an engine name to one placeholder (`<EXE>`, `<VEXE>`), and
`canonical_all()` runs it through the baseline in `setUpClass`, so a file recorded on either platform compares on both.
Both names come from `setup.EXE` and `setup.VEXE` with the extension stripped, so they cannot drift away from setup.
`normalize()` now calls `canonical()`.

`#973`'s rule is kept: only the whole name matches, so `strata-device.exe`, `strata-x.log` and `strata-vision-bench` are
left alone. `tools/test_setup_golden.json` is untouched, so nothing about what the baseline asserts changes - only how
the two sides are compared.

Left alone: `setup.py`, the baseline JSON, every other test.

### How I checked
Linux, no GPU used, `python tools/test_setup_golden.py`:
- before, on a clean checkout of `origin/main` (`296b59d2`): `FAILED (failures=8)` - 5 in `test_yes`, 3 in
  `test_enter_for_every_question` (the other two UD-Q4_K_XL profiles are skipped by the existing `ram < 48` guard);
- after: `OK`, 4 tests;
- the mapping is symmetric: `"<T>/engine/strata-vision.exe"` and `"<T>/engine/strata-vision"` both reduce to
  `"<T>/engine/<VEXE>"`, as do `strata.exe` and `strata` to `<EXE>`, so Windows - where both sides already agree -
  behaves as it did;
- the negatives hold: `strata-device.exe`, `strata-x.log`, `strata-vision-bench` and an already-replaced `<EXE>` come
  back unchanged.
- every other `tools/test_setup_*.py` and `tools/test_strata_mcp.py` passes as it did before the change.

### Not tested
- Windows. There is no Windows machine here. On Windows `canonical()` maps `strata-vision.exe` to `<VEXE>` on both
  sides of the compare, which is what `normalize()` already did to the live side, so the change should be a no-op there
  - but that is reasoning, not a run.
- A baseline recorded on Linux. `--write` would produce one; I reverted it rather than commit a platform-specific
  baseline.

## Extra Notes
- `tools/test_setup_engine_hash.py` fails 2 tests on `main`, unrelated to this and left as it is:
  `AssertionError: SystemExit not raised` in `test_a_first_install_stops_because_there_is_nothing_to_fall_back_to`, and
  `AssertionError: 'keeping the engine that is installed' not found in ''` in
  `test_updating_keeps_the_installed_engine_instead_of_stopping`. Both are `get_prebuilt()`'s SHA-256 refusal. Worth its
  own issue.
- Until this lands, `python tools/test_setup_golden.py` reports those 8 failures on #2041's branch too, which makes that
  PR's "tests pass" line need a caveat it would rather not have.

本站相关内容

相关页面的快捷入口。