#6405 [Engine] 0.2.17: on a 3-card layer split the expert cache shrinks from 5,940 to 3,551 slots (10.40 → 6.19 GiB) at the same VRAM footprint

open · @wulin198550 · 0 comentarios · En GitHub

Setup & installMulti-GPUNVIDIA / CUDA

Descripción

<html>
<body>
<!--StartFragment--><h2 style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; font: 700 19px / 28px -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; margin: 32px 0px 16px; color: rgb(15, 17, 21); letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Problem</h2><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Upgrading 0.2.16 (engine 0.1.62) → 0.2.17 (engine 0.1.63) on the same machine, model and config costs 4.21 GiB of expert cache:<span> </span><strong style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; font-weight: 600;">5,940 slots (10.40 GiB) → 3,551 slots (6.19 GiB)</strong>. The VRAM the cards use is the same in both versions (within 2 MiB per card), so the space did not go to another buffer that the log names; the cache simply gets less.</p><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The measured expert-cache hit rate falls with it, and prefill on long prompts is the case that suffers.</p><div class="_tableScroll_13jw0_199 _tableFill_13jw0_252" style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; max-width: 100%; overflow-x: auto; overscroll-behavior-x: contain; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">
Version | Expert cache | VRAM used, GPU0 / GPU1 / GPU2
-- | -- | --
0.2.16 (engine 0.1.62) | 5,940 slots, 10.40 GiB | 15,329 / 15,309 / 23,681 MiB
0.2.17 (engine 0.1.63) | 3,551 slots, 6.19 GiB | 15,331 / 15,311 / 23,683 MiB

</div><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">All five runs are byte-identical in VRAM (15,331 / 15,311 / 23,683 MiB) and in slot count, and the engine log's<span> </span><code style="corner-shape: superellipse(1.5); text-autospace: no-autospace; --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; display: inline-flex; align-items: center; box-sizing: border-box; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; line-height: 19px; font-optical-sizing: auto; font-size-adjust: none; font-kerning: auto; font-feature-settings: normal; font-variation-settings: normal; font-language-override: normal; font-family: &quot;SF Mono&quot;, &quot;JetBrains Mono&quot;, &quot;Fira Code&quot;, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, &quot;SF Mono&quot;, &quot;JetBrains Mono&quot;, &quot;Fira Code&quot;, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;; background-color: rgb(250, 250, 250); border: 0.666667px solid rgba(0, 0, 0, 0.04); border-radius: 8px; padding: 0px 5px; font-size: 0.875em !important;">engine started</code><span> </span>timestamp changes on each run (verified), so the config is being read.</p><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Things that did<span> </span><strong style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; font-weight: 600;">not</strong><span> </span>fix it and are probably not related:<span> </span><code style="corner-shape: superellipse(1.5); text-autospace: no-autospace; --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; display: inline-flex; align-items: center; box-sizing: border-box; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; line-height: 19px; font-optical-sizing: auto; font-size-adjust: none; font-kerning: auto; font-feature-settings: normal; font-variation-settings: normal; font-language-override: normal; font-family: &quot;SF Mono&quot;, &quot;JetBrains Mono&quot;, &quot;Fira Code&quot;, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, &quot;SF Mono&quot;, &quot;JetBrains Mono&quot;, &quot;Fira Code&quot;, Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;; background-color: rgb(250, 250, 250); border: 0.666667px solid rgba(0, 0, 0, 0.04); border-radius: 8px; padding: 0px 5px; font-size: 0.875em !important;">--prefill auto:65536</code>,<span> </span><code style="corner-shape: superellipse(1.5); text-autospace: no-autospace; --dsw-eleva

En el sitio

Enlaces a install, modelos, releases.