Pull requests / #2879
#2879 tools: read CMakeLists.txt as UTF-8, not through the OS code page (#2792)
open · @ctr-137 · 0 コメント · GitHub で見る
BenchmarksSetup & installWindowsLinux
本文
`tools/strata_bench.py:564` read this install's own `CMakeLists.txt` with no `encoding=`, so the decode went
through `locale.getpreferredencoding()`. That is GBK or cp932 on a Chinese or Japanese Windows, where the
file's UTF-8 BOM is not a valid sequence - the third byte `0xbf` - and `BENCH.bat` died before any
measurement:
```
File "tools/strata_bench.py", line 564, in project_version
m = re.search(r"project\(strata VERSION ([\d.]+)", (ROOT / "CMakeLists.txt").read_text())
UnicodeDecodeError: 'gbk' codec can't decode byte 0xbf in position 2: illegal multibyte sequence
```
`setup.py` reads the same file with `encoding="utf-8"`, so this is an omission here rather than an
intentional locale dependency.
A cp1252 machine decodes the BOM to mojibake and says nothing, which is why only those two locales ever saw
it - and why this needs a test rather than a reproducer.
### What I changed
`tools/strata_bench.py`: read it as `utf-8-sig`. That matches how this file already reads its other repo and
config files (lines 657, 688, 971), and it strips the BOM the file actually has rather than leaving a
`` in front of the string the regex scans.
### What I left alone
- `/proc/meminfo` and `/proc/cpuinfo` are still read without `encoding=`. They are the Linux-only branch
and their content is ASCII whatever the locale, so I did not touch them.
- `project_version()` still catches only `OSError`. A repo file that is unreadable for any other reason
still reaches the top of `main()` as a traceback and leaves a started server behind, which the report also
notes. `UnicodeDecodeError` is a `ValueError`, so widening that except would cover it - but what the
bench should do with an unreadable install is a separate decision and I did not want to bundle it here.
### How I checked
Three tests in `tools/test_strata_bench.py`, no GPU or server needed:
- the encoding `project_version()` passes is captured and has to be a UTF-8 one, so a future call that
silently goes back to the code page fails;
- the file really is undecodable as GBK, which pins why the BOM is the thing to survive and documents the
reported failure;
- the version it returns is the same as a strict `utf-8-sig` read of the file.
With the fix reverted the first of them fails (`encoding=''`), with it all 20 tests in
`tools/test_strata_bench.py` pass (`python -m unittest tools.test_strata_bench`).
The reporter's workaround (`set PYTHONUTF8=1`) still works and is no longer needed.
Fixes #2792.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
関連リンク
インストール・モデル・リリースへの站内リンク。