Issues / #477

#477 Persist the runtime expert heat map across restarts; optionally select a profile per working directory

closed · @AXKore · 2 comentarios · En GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsWindows

Descripción

What the engine already does
At startup, GPU expert residency is seeded from the shipped static profile (data/expert-profile.bin), then refined at runtime by --adapt-every as real token traffic is observed.
Measured on my side (RTX 4090 24 GB, IQ3_S, 262K context): the decode cache hit rate climbs 63% → 73% → 85% within the first requests, and long agent sessions then hold a much higher sustained decode rate (65-75 tok/s, MTP draft acceptance 65-84%).
The gap
Everything learned since engine start lives only in RAM. On restart, the engine falls back to the generic shipped profile, and the first minutes of every session are spent re-learning my usage pattern from scratch.

Proposal — three levels, each independently useful
Persist the learned residency. Save the runtime heat map on clean exit (or periodically) and reload it at start instead of the shipped profile — e.g. --expert-profile-save <path> paired with the existing --expert-profile <path>. A cold start becomes warm.
Per-context selection. The engine already accepts a profile file at start, so launchers could pick one per working directory (code vs prose vs a specific codebase — the token distribution of a project is a signature, and different projects heat different experts). Only the save side from (1) is missing to make this possible.
Later: opt-in sharing/blending of profiles. They are small (KB-scale), so per-user or per-domain profiles ("Python coder", "FR writer"…) could be exchanged. A profile is a usage fingerprint, though, so this should stay strictly opt-in.
Why it fits the current design
The load path exists today; the runtime measurement exists today. This is only save + selection built around them.
Staleness risk is minimal: --adapt-every still refines at runtime, so an outdated profile is simply a better-than-generic prior, never a wrong answer.
Great fit for agent workloads (Claude Code and friends), where engines get restarted often and the first minutes matter.
Note
Keeping the save feature off by default (or local-only) seems right, since profiles reflect what a user works on.

Environment: engine 0.1.34, Windows 11, RTX 4090, Ryzen 9 5950X, 128 GB RAM, IQ3_S packed install.

En el sitio

Enlaces a install, modelos, releases.