Pull requests / #2935

#2935 calibrate: survive an engine dying mid-measurement; adaptive gate: say where the copies stopped (#1337, #2188)

open · @dsenchankau · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDADocumentation

Beschreibung

# PR body - replace GitHub's pre-filled body with everything below this line

## Title

Issue: extends #1337 (not Resolves - that shipped in 0.1.41); diagnostics for #2188 (still open)

## Summary

--calibrate tries several settings in turn and restarts the engine for each
one.  Since 0.1.41, an engine that refuses to start only skips its own
setting (#1337) - everything measured until then is kept.  But an engine
that starts fine and then dies during its measurement still killed the whole
calibration.  A real run on a UD-Q4_K_XL PC: --adapt-swaps 80 measured fine,
the --adapt-swaps 160 engine then died at the adaptive tier's copy gate
(#2188's class), and the --pcie-frac / --spec-min-p winners from the first
20 minutes were lost with it.  After this change the dead engine drops only
its own candidate: the earlier settings survive, and the winner is picked
from the settings that did measure.

## What changed

tools/calibrate.py

- If the engine dies during a measurement of the CPU worker counts (step 4),
  the expert tier (step 5) or the shared-expert fork (step 6), only that
  setting is dropped.  The reason is recorded in the report's failed_starts
  list, exactly like a failed start already is, and the calibration carries
  on.
- The winner is still picked from the settings that measured, by the same
  rule as always (it must beat the default by more than 3%).  In the repro
  above, 80 swaps measured 10% over the default and is still written, even
  though 160 died.  One dying never pushes the answer back to the default;
  the only way a setting is not written is that it, or the default setting
  itself, was not measured - then the product default simply stays.
- Steps 4, 5 and 6 now share one restart-and-measure function
  (measure_candidate).  Step 6 (AMD only) had no protection against a death
  during its measurement before, and died the same way.
- What does not change: a failed start behaves exactly as in 0.1.41; the
  first start (steps 1-3) still fails the whole run, because nothing is
  measured yet; and a calibration where no engine dies behaves exactly as
  before.

src/core/verify.cpp

- When the engine times out waiting for the adaptive tier's expert copies
  (#2188's error), the message now says how far the copies actually got:
  the layer the gate reached, its counter, and what the copy stream itself
  reported - and it says plainly that releasing the GPU waits (the #267
  fix) drains the window's own streams but never the tier's copy stream.
  With STRATA_VERIFY_TRACE=1 it also dumps the window's layer-by-layer
  breadcrumbs.  This code only runs in a wait that has already failed, so a
  run that never times out is untouched.

tools/test_calibrate.py

- The fake test engine can now die during its own first generation (dies=).
  Three new tests: the tier candidate that dies loses while the faster
  80-swap candidate is still chosen; a worker count that dies is dropped and
  the run still measures the rest, including the tier; a fork candidate that
  dies is dropped and the run returns normally.

## Extra Notes

- How this helps the open #2188 AMD/WDDM report: the new numbers tell the
  two remaining causes apart.  If the counter climbs while the copy stream's
  own value lags behind, copies were queued but never submitted to the GPU
  (the WDDM flush problem, the class 0.1.44 fixed for NVIDIA).  If both stop
  at the same layer and a later request fails with "layer N never rang", it
  is the GPU itself stopping.  A rerun with STRATA_VERIFY_TRACE=1 prints the
  last layer the copy stream reached.
- How this was checked, on Ubuntu 24.04, i7-13700F, RTX 4080 SUPER (sm_89),
  CUDA 13.4 (nvcc V13.4.92): python3 tools/test_calibrate.py passes 30/30
  (27 before this change); a CUDA build with -DSTRATA_BUILD_TESTS=ON passes
  ctest 142/143 - expert_multi_test skips itself without AVX-512, and
  ple_parity needs the bench/micro fixture files, which a clean checkout
  does not have (docs/AMD_HIP.md says the same).
- Not tested: a HIP build - no AMD hardware here.  The verify.cpp block is
  plain C++ shared by the CUDA and HIP builds, and the SYCL tree has no
  adaptive gate at all, so only CUDA/HIP compile this function.
- The new C++ message cannot be covered by a ctest from a clean checkout:
  reaching it needs the real #2188 stop (GPU, adaptive tier, overlap on).
  The existing STRATA_TEST_VERIFY_STALL hook stops the window's own flags,
  not the gate.

Mehr auf der Site

Links zu Install, Modellen, Releases.