Pull requests / #2935
#2935 calibrate: survive an engine dying mid-measurement; adaptive gate: say where the copies stopped (#1337, #2188)
open · @dsenchankau · 0 comentários · No GitHub
BenchmarksAMD / HIPNVIDIA / CUDADocumentation
Descrição
# PR body - replace GitHub's pre-filled body with everything below this line ## Title Issue: extends #1337 (not Resolves - that shipped in 0.1.41); diagnostics for #2188 (still open) ## Summary --calibrate tries several settings in turn and restarts the engine for each one. Since 0.1.41, an engine that refuses to start only skips its own setting (#1337) - everything measured until then is kept. But an engine that starts fine and then dies during its measurement still killed the whole calibration. A real run on a UD-Q4_K_XL PC: --adapt-swaps 80 measured fine, the --adapt-swaps 160 engine then died at the adaptive tier's copy gate (#2188's class), and the --pcie-frac / --spec-min-p winners from the first 20 minutes were lost with it. After this change the dead engine drops only its own candidate: the earlier settings survive, and the winner is picked from the settings that did measure. ## What changed tools/calibrate.py - If the engine dies during a measurement of the CPU worker counts (step 4), the expert tier (step 5) or the shared-expert fork (step 6), only that setting is dropped. The reason is recorded in the report's failed_starts list, exactly like a failed start already is, and the calibration carries on. - The winner is still picked from the settings that measured, by the same rule as always (it must beat the default by more than 3%). In the repro above, 80 swaps measured 10% over the default and is still written, even though 160 died. One dying never pushes the answer back to the default; the only way a setting is not written is that it, or the default setting itself, was not measured - then the product default simply stays. - Steps 4, 5 and 6 now share one restart-and-measure function (measure_candidate). Step 6 (AMD only) had no protection against a death during its measurement before, and died the same way. - What does not change: a failed start behaves exactly as in 0.1.41; the first start (steps 1-3) still fails the whole run, because nothing is measured yet; and a calibration where no engine dies behaves exactly as before. src/core/verify.cpp - When the engine times out waiting for the adaptive tier's expert copies (#2188's error), the message now says how far the copies actually got: the layer the gate reached, its counter, and what the copy stream itself reported - and it says plainly that releasing the GPU waits (the #267 fix) drains the window's own streams but never the tier's copy stream. With STRATA_VERIFY_TRACE=1 it also dumps the window's layer-by-layer breadcrumbs. This code only runs in a wait that has already failed, so a run that never times out is untouched. tools/test_calibrate.py - The fake test engine can now die during its own first generation (dies=). Three new tests: the tier candidate that dies loses while the faster 80-swap candidate is still chosen; a worker count that dies is dropped and the run still measures the rest, including the tier; a fork candidate that dies is dropped and the run returns normally. ## Extra Notes - How this helps the open #2188 AMD/WDDM report: the new numbers tell the two remaining causes apart. If the counter climbs while the copy stream's own value lags behind, copies were queued but never submitted to the GPU (the WDDM flush problem, the class 0.1.44 fixed for NVIDIA). If both stop at the same layer and a later request fails with "layer N never rang", it is the GPU itself stopping. A rerun with STRATA_VERIFY_TRACE=1 prints the last layer the copy stream reached. - How this was checked, on Ubuntu 24.04, i7-13700F, RTX 4080 SUPER (sm_89), CUDA 13.4 (nvcc V13.4.92): python3 tools/test_calibrate.py passes 30/30 (27 before this change); a CUDA build with -DSTRATA_BUILD_TESTS=ON passes ctest 142/143 - expert_multi_test skips itself without AVX-512, and ple_parity needs the bench/micro fixture files, which a clean checkout does not have (docs/AMD_HIP.md says the same). - Not tested: a HIP build - no AMD hardware here. The verify.cpp block is plain C++ shared by the CUDA and HIP builds, and the SYCL tree has no adaptive gate at all, so only CUDA/HIP compile this function. - The new C++ message cannot be covered by a ctest from a clean checkout: reaching it needs the real #2188 stop (GPU, adaptive tier, overlap on). The existing STRATA_TEST_VERIFY_STALL hook stops the window's own flags, not the gate.
No site
Links install, modelos, releases.