#6571 [Engine] Merge Ryu's SM x VRAM profile tables into official CUDA Inkling-2
open · @zapabob · 0 comments · View on GitHub
Description
## Problem Official Inkling-2 Mini (v0.2.19) has no automatic profile selection: the CUDA path queries SM arch and VRAM but picks kernel tuning constants manually. On measured cards (V100 sm_70 32 GB, RTX 4070 sm_89 12 GB) the best prefill chunk, KV strategy and expert placement depend on both the SM class and the VRAM residency regime, and hard-coding one set regresses the other card. ## Result A pure-C++ two-axis (SM arch x VRAM regime) profile table is included into the official CUDA Inkling-2 path. It is header-only, GPU-free, and follows Strata's measured-only / recommend-never-force rule: unmeasured GPUs take conservative Generic defaults, and explicit CLI flags / env overrides still win at the caller. ## Approach - Add `engine/include/strata/inkling2/profile_rules.hpp` (measured sm_70/sm_89 + Generic tables). - `inkling2_loop.hpp`: include the table with credit. - `ink_cuda.cuh`: add `current_arch_profile()` mapping `current_sm_arch()` to the measured ArchProfile. - Credit to kuraneko1 (_ryu15_), kuraneko1/Strata (MIT), inlined in the code comments. ## Validation - Full CMake configure + build of the official `strata-inkling` target with CUDA 13.5 / sm_120: exit 0, `strata-inkling.exe` produced, `--help` works. - The table itself is header-only and has a CPU-only unit test in the source fork (`inkling2_profile_rules_test.cpp`). ## Limits - I did not run real-model inference for this PR; only build + `--help` were verified on this machine (RTX 5060 Ti 16 GB). Real GGUF measurement on the two measured cards (V100, RTX 4070) is still needed before treating any default as changed. No defaults are changed here; the table is opt-in at the caller. This is a focused integration, not a feature proposal: it ports an existing MIT implementation (kuraneko1/Strata, itself a fork of this repo) into the official CUDA path. See issue scope if a larger refactor is desired.
Related on this site
Quick links to related pages.