改动 / #3357
#3357 [Other] setup: lock the n-gram table in RAM only when the experts still fit
open · @dsenchankau · 0 评论 · 去 GitHub 看
BenchmarksSetup & installModels & quantsWindowsLinux
说明
### Problem and result Since v0.2.0-beta setup sometimes writes `--ple-io ram` on SSD PCs: the model's n-gram table is locked in RAM so a new prompt does not wait for its table rows from the disk. It only does this when a RAM check says there is room, but for the two 4-bit Unsloth files (`UD-IQ4_XS`, `UD-Q4_K_XL`) that check used the wrong number. It counted the model with its minimum requirement: the RAM it needs when part of its experts live on disk. On those same machines setup had just chosen the mode that keeps every expert in RAM, so both together do not fit. I hit this myself on my 96 GB machine (`ram_gb()` sees 94) with UD-Q4_K_XL: fully loaded, with every expert resident, RAM sits at 91.5 GB — there is plainly no room for the 29 GB table on top. The engine then streamed experts from the model files and decode dropped from roughly 40 t/s to 15-20 t/s. A 96 GB machine gets the wrong flag with both 4-bit files, a 128 GB machine with UD-Q4_K_XL. After this change those machines read the table from disk and keep the experts in RAM. The rotating-disk version of the rule had the same wrong number and pushed the experts onto the slowest storage possible, so I fixed it the same way. ### Approach Both rules now work out what the install puts in RAM the same way the code above them does when it picks the all-in-RAM mode: the whole expert set plus the usual allowance beside it. If the table and the existing spare do not fit on top of that, setup leaves the table on disk. Nothing changed for the other model families, their numbers were already right, and the Windows path never reaches this code. The bigger Unsloth files move by under a GB, which changes nothing. Machines that were already set up by v0.2.0-beta keep the flag they got: run setup again, or drop `--ple-io`, `ram` from the config's args by hand. ### Validation (Linux, Python 3.12.3) **Passed** - Unit tests (mocked PC, no GPU needed): ``` python3 -m unittest tools.test_setup_choices # 55 tests, all OK python3 -m unittest tools.test_setup_risk tools.test_setup_menu tools.test_setup_sycl # all OK python3 -m unittest tools.test_setup_qwen36 tools.test_setup_gyro # all OK ``` The new tests cover both 4-bit files on machines from 32 to 192 GB, on an SSD and on a hard disk, and they finally show the rule the big sizes' real 54 GB table (before, every test took the 29 GB fallback). I also checked the tests actually catch the bug: copied onto the commit before the fix they fail (UD-IQ4_XS on a 96 GB machine wrongly gets `--ple-io ram`), and they pass here. - Ran `setup.sh` on my own PC (96 GB) for both UD-IQ4_XS and UD-Q4_K_XL. The configs they write no longer have the wrong `--ple-io ram` entry, and the rest of the config is unchanged — still the mode that keeps every expert in RAM. With the fixed config decode is back to roughly 40 t/s, against 15-20 t/s before. My machine, everyday use, not a formal benchmark run. - Engine test suite (C++, untouched by this PR, ran it as a regression check): ``` ctest --test-dir build-tests --output-on-failure ``` 151 of 157 passed (96%), exactly the same as the latest master commit on this machine. Same six failures with and without my change (`ple_parity`, `ggml_tq_overlay`, `native_hadamard_test`, `tq_cpu_parity`, `tq_cpu_parity_avx2`, `isa_guard_check`). Same one skip (`expert_multi_test`). 215 s. **Failed** Nothing among the commands above. **Not run** - The other quant sizes on real hardware — the RAM check change is the same code for all of them, but I only ran mine. - Windows and WSL — the code does not run there. - The golden-config suite fails the same on master in our fork (a fork-local difference), so I did not count it either way. ### Limits The check is still an estimate.
本站相关内容
相关页面的快捷入口。